2024-04-24T22:14:52.655 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-24T22:14:52.655 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/yuriw-2024-04-24_21:25:53-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7672255 --owner scheduled_yuriw@teuthology --name yuriw-2024-04-24_21:25:53-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} -- /home/teuthworker/archive/yuriw-2024-04-24_21:25:53-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7672255/orig.config.yaml 2024-04-24T22:14:52.703 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T22:14:52.787 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-24_21:25:53-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7672255 branch: wip-yuriw4-testing-20240423.151325-quincy description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} email: null first_in_suite: false job_id: '7672255' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: yuriw-2024-04-24_21:25:53-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuriw4-testing-20240423.151325-quincy ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: none client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuriw4-testing-20240423.151325-quincy sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 9954 sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: wip-yuriw4-testing-20240423.151325-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_330067dbce47655d1f762ef14d546d2fde16a6a9/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 targets: smithi080.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFtIISksWdrVRHufao/k3yWyq3S/2VFVLmirMvq9bK3yYzjhKiiias2g1SEIIOlD4G0INN8vwRO7Xj2+HQW6lo0= smithi134.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBHEp9XcSD+LRpP+1zpkogwKIriYSdDjcXeYzLZCqzOirEGLV4Wd+CCGg/fGq3EI+YNtR9rEpDMqf2Rhx3KfpQQ= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_IMAGE_MODE: snapshot MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_21:25:53 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-24T22:14:52.787 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_330067dbce47655d1f762ef14d546d2fde16a6a9/qa; will attempt to use it 2024-04-24T22:14:52.788 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_330067dbce47655d1f762ef14d546d2fde16a6a9/qa/tasks 2024-04-24T22:14:52.788 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-24T22:14:52.789 INFO:teuthology.task.internal:Checking packages... 2024-04-24T22:14:52.814 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '330067dbce47655d1f762ef14d546d2fde16a6a9' 2024-04-24T22:14:52.815 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-24T22:14:52.815 INFO:teuthology.packaging:ref: None 2024-04-24T22:14:52.815 INFO:teuthology.packaging:tag: None 2024-04-24T22:14:52.815 INFO:teuthology.packaging:branch: wip-yuriw4-testing-20240423.151325-quincy 2024-04-24T22:14:52.815 INFO:teuthology.packaging:sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T22:14:52.815 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuriw4-testing-20240423.151325-quincy 2024-04-24T22:14:52.958 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-915.g330067db 2024-04-24T22:14:52.960 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-24T22:14:52.967 INFO:teuthology.task.internal:no buildpackages task found 2024-04-24T22:14:52.967 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-24T22:14:52.974 INFO:teuthology.task.internal:Saving configuration 2024-04-24T22:14:52.992 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-24T22:14:52.999 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T22:14:53.025 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi080.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-24_21:25:53-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7672255', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 22:09:44.711597', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFtIISksWdrVRHufao/k3yWyq3S/2VFVLmirMvq9bK3yYzjhKiiias2g1SEIIOlD4G0INN8vwRO7Xj2+HQW6lo0='} 2024-04-24T22:14:53.049 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-24_21:25:53-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7672255', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 22:09:44.709615', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBHEp9XcSD+LRpP+1zpkogwKIriYSdDjcXeYzLZCqzOirEGLV4Wd+CCGg/fGq3EI+YNtR9rEpDMqf2Rhx3KfpQQ='} 2024-04-24T22:14:53.049 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-24T22:14:53.056 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-04-24T22:14:53.056 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-04-24T22:14:53.056 INFO:teuthology.run_tasks:Running task console_log... 2024-04-24T22:14:53.146 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa36d7b9160>, signals=[15]) 2024-04-24T22:14:53.146 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-24T22:14:53.152 INFO:teuthology.task.internal:Opening connections... 2024-04-24T22:14:53.152 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi080.front.sepia.ceph.com 2024-04-24T22:14:53.154 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T22:14:53.228 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi134.front.sepia.ceph.com 2024-04-24T22:14:53.229 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T22:14:53.302 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-24T22:14:53.310 DEBUG:teuthology.orchestra.run.smithi080:> uname -m 2024-04-24T22:14:53.331 INFO:teuthology.orchestra.run.smithi080.stdout:x86_64 2024-04-24T22:14:53.331 DEBUG:teuthology.orchestra.run.smithi080:> cat /etc/os-release 2024-04-24T22:14:53.390 INFO:teuthology.orchestra.run.smithi080.stdout:NAME="CentOS Stream" 2024-04-24T22:14:53.390 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION="8" 2024-04-24T22:14:53.390 INFO:teuthology.orchestra.run.smithi080.stdout:ID="centos" 2024-04-24T22:14:53.390 INFO:teuthology.orchestra.run.smithi080.stdout:ID_LIKE="rhel fedora" 2024-04-24T22:14:53.390 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION_ID="8" 2024-04-24T22:14:53.390 INFO:teuthology.orchestra.run.smithi080.stdout:PLATFORM_ID="platform:el8" 2024-04-24T22:14:53.390 INFO:teuthology.orchestra.run.smithi080.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-24T22:14:53.390 INFO:teuthology.orchestra.run.smithi080.stdout:ANSI_COLOR="0;31" 2024-04-24T22:14:53.390 INFO:teuthology.orchestra.run.smithi080.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-24T22:14:53.391 INFO:teuthology.orchestra.run.smithi080.stdout:HOME_URL="https://centos.org/" 2024-04-24T22:14:53.391 INFO:teuthology.orchestra.run.smithi080.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T22:14:53.391 INFO:teuthology.orchestra.run.smithi080.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-24T22:14:53.391 INFO:teuthology.orchestra.run.smithi080.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T22:14:53.391 INFO:teuthology.lock.ops:Updating smithi080.front.sepia.ceph.com on lock server 2024-04-24T22:14:53.415 DEBUG:teuthology.orchestra.run.smithi134:> uname -m 2024-04-24T22:14:53.437 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64 2024-04-24T22:14:53.437 DEBUG:teuthology.orchestra.run.smithi134:> cat /etc/os-release 2024-04-24T22:14:53.498 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="CentOS Stream" 2024-04-24T22:14:53.499 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="8" 2024-04-24T22:14:53.499 INFO:teuthology.orchestra.run.smithi134.stdout:ID="centos" 2024-04-24T22:14:53.499 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE="rhel fedora" 2024-04-24T22:14:53.499 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="8" 2024-04-24T22:14:53.499 INFO:teuthology.orchestra.run.smithi134.stdout:PLATFORM_ID="platform:el8" 2024-04-24T22:14:53.499 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-24T22:14:53.499 INFO:teuthology.orchestra.run.smithi134.stdout:ANSI_COLOR="0;31" 2024-04-24T22:14:53.499 INFO:teuthology.orchestra.run.smithi134.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-24T22:14:53.499 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://centos.org/" 2024-04-24T22:14:53.499 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T22:14:53.499 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-24T22:14:53.499 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T22:14:53.500 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2024-04-24T22:14:53.523 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-24T22:14:53.533 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-24T22:14:53.538 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-24T22:14:53.538 DEBUG:teuthology.orchestra.run.smithi080:> test '!' -e /home/ubuntu/cephtest 2024-04-24T22:14:53.540 DEBUG:teuthology.orchestra.run.smithi134:> test '!' -e /home/ubuntu/cephtest 2024-04-24T22:14:53.563 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-24T22:14:53.569 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-24T22:14:53.569 DEBUG:teuthology.orchestra.run.smithi080:> test -z $(ls -A /var/lib/ceph) 2024-04-24T22:14:53.600 DEBUG:teuthology.orchestra.run.smithi134:> test -z $(ls -A /var/lib/ceph) 2024-04-24T22:14:53.631 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-24T22:14:53.707 INFO:teuthology.run_tasks:Running task kernel... 2024-04-24T22:14:53.731 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-24T22:14:53.731 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-24T22:14:53.732 DEBUG:teuthology.orchestra.run.smithi080:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T22:14:53.732 DEBUG:teuthology.orchestra.run.smithi134:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T22:14:53.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:14:53.759 DEBUG:teuthology.orchestra.run.smithi080:> uname -r 2024-04-24T22:14:53.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:14:53.760 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-04-24T22:14:53.823 INFO:teuthology.orchestra.run.smithi080.stdout:4.18.0-489.el8.x86_64 2024-04-24T22:14:53.823 INFO:teuthology.task.kernel:Running kernel on smithi080: 4.18.0-489.el8.x86_64 2024-04-24T22:14:53.824 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum install -y kernel 2024-04-24T22:14:53.830 INFO:teuthology.orchestra.run.smithi134.stdout:4.18.0-489.el8.x86_64 2024-04-24T22:14:53.830 INFO:teuthology.task.kernel:Running kernel on smithi134: 4.18.0-489.el8.x86_64 2024-04-24T22:14:53.830 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum install -y kernel 2024-04-24T22:14:57.296 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - AppStream 30 MB/s | 28 MB 00:00 2024-04-24T22:14:57.570 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 8 - AppStream 26 MB/s | 28 MB 00:01 2024-04-24T22:15:03.770 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - BaseOS 7.5 MB/s | 10 MB 00:01 2024-04-24T22:15:04.028 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 8 - BaseOS 7.2 MB/s | 10 MB 00:01 2024-04-24T22:15:06.721 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - Extras 13 kB/s | 18 kB 00:01 2024-04-24T22:15:07.295 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 8 - Extras 11 kB/s | 18 kB 00:01 2024-04-24T22:15:07.671 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - Extras common packages 9.2 kB/s | 7.7 kB 00:00 2024-04-24T22:15:07.914 INFO:teuthology.orchestra.run.smithi134.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-04-24T22:15:08.112 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 8 - Extras common packages 11 kB/s | 7.7 kB 00:00 2024-04-24T22:15:08.330 INFO:teuthology.orchestra.run.smithi080.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2024-04-24T22:15:08.881 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 19 MB/s | 16 MB 00:00 2024-04-24T22:15:10.664 INFO:teuthology.orchestra.run.smithi080.stdout:Extra Packages for Enterprise Linux 7.3 MB/s | 16 MB 00:02 2024-04-24T22:15:12.723 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 453 kB/s | 24 kB 00:00 2024-04-24T22:15:14.456 INFO:teuthology.orchestra.run.smithi080.stdout:lab-extras 573 kB/s | 24 kB 00:00 2024-04-24T22:15:15.795 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-24T22:15:15.796 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-24T22:15:15.907 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:15:15.908 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:15:15.908 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-04-24T22:15:15.908 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:15:15.908 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-24T22:15:15.909 INFO:teuthology.orchestra.run.smithi134.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-24T22:15:15.909 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-04-24T22:15:15.909 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-24T22:15:15.909 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-24T22:15:15.909 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:15:15.909 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-24T22:15:15.909 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:15:15.909 INFO:teuthology.orchestra.run.smithi134.stdout:Install 3 Packages 2024-04-24T22:15:15.909 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:15:15.910 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 90 M 2024-04-24T22:15:15.910 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 96 M 2024-04-24T22:15:15.910 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-24T22:15:17.008 INFO:teuthology.orchestra.run.smithi134.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 10 MB/s | 10 MB 00:01 2024-04-24T22:15:17.519 INFO:teuthology.orchestra.run.smithi080.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-24T22:15:17.519 INFO:teuthology.orchestra.run.smithi080.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-24T22:15:17.627 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:15:17.628 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:15:17.628 INFO:teuthology.orchestra.run.smithi080.stdout: Package Architecture Version Repository Size 2024-04-24T22:15:17.628 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:15:17.628 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-04-24T22:15:17.628 INFO:teuthology.orchestra.run.smithi080.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-24T22:15:17.628 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-04-24T22:15:17.628 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-24T22:15:17.629 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-24T22:15:17.629 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:15:17.629 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-04-24T22:15:17.629 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:15:17.629 INFO:teuthology.orchestra.run.smithi080.stdout:Install 3 Packages 2024-04-24T22:15:17.629 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:15:17.629 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 90 M 2024-04-24T22:15:17.629 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 96 M 2024-04-24T22:15:17.630 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-04-24T22:15:17.841 INFO:teuthology.orchestra.run.smithi134.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 19 MB/s | 36 MB 00:01 2024-04-24T22:15:18.325 INFO:teuthology.orchestra.run.smithi134.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 19 MB/s | 44 MB 00:02 2024-04-24T22:15:18.325 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:15:18.325 INFO:teuthology.orchestra.run.smithi134.stdout:Total 37 MB/s | 90 MB 00:02 2024-04-24T22:15:18.657 INFO:teuthology.orchestra.run.smithi080.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 12 MB/s | 10 MB 00:00 2024-04-24T22:15:19.008 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-24T22:15:19.433 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-24T22:15:19.433 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-24T22:15:19.515 INFO:teuthology.orchestra.run.smithi080.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 22 MB/s | 36 MB 00:01 2024-04-24T22:15:19.899 INFO:teuthology.orchestra.run.smithi080.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 21 MB/s | 44 MB 00:02 2024-04-24T22:15:19.899 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:15:19.899 INFO:teuthology.orchestra.run.smithi080.stdout:Total 40 MB/s | 90 MB 00:02 2024-04-24T22:15:20.584 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-04-24T22:15:20.676 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-24T22:15:20.699 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-24T22:15:20.994 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-04-24T22:15:20.994 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-04-24T22:15:22.238 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-04-24T22:15:22.265 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-04-24T22:15:23.470 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-24T22:15:23.977 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-24T22:15:25.030 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-04-24T22:15:25.580 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-24T22:15:27.590 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-24T22:15:27.977 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-24T22:15:29.136 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-24T22:15:29.579 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-24T22:15:31.409 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-24T22:15:31.693 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-24T22:15:33.021 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-24T22:15:33.328 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-24T22:15:53.584 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-24T22:15:53.607 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-24T22:15:53.942 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-24T22:15:53.942 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-24T22:15:53.942 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-24T22:15:54.453 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-24T22:15:54.453 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:15:54.453 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-24T22:15:54.453 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-24T22:15:54.453 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-24T22:15:54.453 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:15:54.454 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:15:54.623 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-04-24T22:15:54.841 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-24T22:15:54.841 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-24T22:15:54.841 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-24T22:15:54.841 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-24T22:15:54.842 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-24T22:15:54.842 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-04-24T22:15:54.842 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi134.front.sepia.ceph.com, path=None, version=distro) 2024-04-24T22:15:54.842 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-04-24T22:15:55.054 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-24T22:15:55.054 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-24T22:15:55.054 DEBUG:teuthology.orchestra.run.smithi134:> sudo rpm -qi grub2-tools 2024-04-24T22:15:55.087 INFO:teuthology.orchestra.run.smithi134.stdout:Name : grub2-tools 2024-04-24T22:15:55.087 INFO:teuthology.orchestra.run.smithi134.stdout:Epoch : 1 2024-04-24T22:15:55.088 INFO:teuthology.orchestra.run.smithi134.stdout:Version : 2.02 2024-04-24T22:15:55.088 INFO:teuthology.orchestra.run.smithi134.stdout:Release : 129.el8 2024-04-24T22:15:55.088 INFO:teuthology.orchestra.run.smithi134.stdout:Architecture: x86_64 2024-04-24T22:15:55.088 INFO:teuthology.orchestra.run.smithi134.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-24T22:15:55.088 INFO:teuthology.orchestra.run.smithi134.stdout:Group : System Environment/Base 2024-04-24T22:15:55.088 INFO:teuthology.orchestra.run.smithi134.stdout:Size : 9265586 2024-04-24T22:15:55.088 INFO:teuthology.orchestra.run.smithi134.stdout:License : GPLv3+ 2024-04-24T22:15:55.088 INFO:teuthology.orchestra.run.smithi134.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-24T22:15:55.088 INFO:teuthology.orchestra.run.smithi134.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-24T22:15:55.088 INFO:teuthology.orchestra.run.smithi134.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-24T22:15:55.088 INFO:teuthology.orchestra.run.smithi134.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-24T22:15:55.088 INFO:teuthology.orchestra.run.smithi134.stdout:Relocations : (not relocatable) 2024-04-24T22:15:55.088 INFO:teuthology.orchestra.run.smithi134.stdout:Packager : CentOS BuildSystem 2024-04-24T22:15:55.088 INFO:teuthology.orchestra.run.smithi134.stdout:Vendor : CentOS 2024-04-24T22:15:55.089 INFO:teuthology.orchestra.run.smithi134.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-24T22:15:55.089 INFO:teuthology.orchestra.run.smithi134.stdout:Summary : Support tools for GRUB. 2024-04-24T22:15:55.089 INFO:teuthology.orchestra.run.smithi134.stdout:Description : 2024-04-24T22:15:55.089 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:15:55.089 INFO:teuthology.orchestra.run.smithi134.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-24T22:15:55.089 INFO:teuthology.orchestra.run.smithi134.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-24T22:15:55.089 INFO:teuthology.orchestra.run.smithi134.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-24T22:15:55.089 INFO:teuthology.orchestra.run.smithi134.stdout:hardware devices. 2024-04-24T22:15:55.089 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:15:55.089 INFO:teuthology.orchestra.run.smithi134.stdout:This subpackage provides tools for support of all platforms. 2024-04-24T22:15:55.090 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-24T22:15:55.090 INFO:teuthology.task.kernel:Updating grub on smithi134 to boot 4.18.0-552.el8.x86_64 2024-04-24T22:15:55.090 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-24T22:15:55.460 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-24T22:15:55.487 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-24T22:15:55.923 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-24T22:15:55.923 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-24T22:15:55.923 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-24T22:15:56.384 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-24T22:15:56.384 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:15:56.384 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-04-24T22:15:56.385 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-24T22:15:56.385 INFO:teuthology.orchestra.run.smithi080.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-24T22:15:56.385 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:15:56.385 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:15:56.534 DEBUG:teuthology.orchestra.run.smithi080:> rpm -q kernel | sort -rV | head -n 1 2024-04-24T22:15:56.754 INFO:teuthology.orchestra.run.smithi080.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-24T22:15:56.755 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-24T22:15:56.755 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-24T22:15:56.755 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-24T22:15:56.755 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-24T22:15:56.755 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-04-24T22:15:56.755 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi080.front.sepia.ceph.com, path=None, version=distro) 2024-04-24T22:15:56.755 DEBUG:teuthology.orchestra.run.smithi080:> rpm -q kernel | sort -rV | head -n 1 2024-04-24T22:15:56.866 INFO:teuthology.orchestra.run.smithi134.stderr:Generating grub configuration file ... 2024-04-24T22:15:56.969 INFO:teuthology.orchestra.run.smithi080.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-24T22:15:56.970 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-24T22:15:56.970 DEBUG:teuthology.orchestra.run.smithi080:> sudo rpm -qi grub2-tools 2024-04-24T22:15:57.003 INFO:teuthology.orchestra.run.smithi080.stdout:Name : grub2-tools 2024-04-24T22:15:57.003 INFO:teuthology.orchestra.run.smithi080.stdout:Epoch : 1 2024-04-24T22:15:57.004 INFO:teuthology.orchestra.run.smithi080.stdout:Version : 2.02 2024-04-24T22:15:57.004 INFO:teuthology.orchestra.run.smithi080.stdout:Release : 129.el8 2024-04-24T22:15:57.004 INFO:teuthology.orchestra.run.smithi080.stdout:Architecture: x86_64 2024-04-24T22:15:57.004 INFO:teuthology.orchestra.run.smithi080.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-24T22:15:57.004 INFO:teuthology.orchestra.run.smithi080.stdout:Group : System Environment/Base 2024-04-24T22:15:57.004 INFO:teuthology.orchestra.run.smithi080.stdout:Size : 9265586 2024-04-24T22:15:57.004 INFO:teuthology.orchestra.run.smithi080.stdout:License : GPLv3+ 2024-04-24T22:15:57.004 INFO:teuthology.orchestra.run.smithi080.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-24T22:15:57.004 INFO:teuthology.orchestra.run.smithi080.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-24T22:15:57.004 INFO:teuthology.orchestra.run.smithi080.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-24T22:15:57.004 INFO:teuthology.orchestra.run.smithi080.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-24T22:15:57.004 INFO:teuthology.orchestra.run.smithi080.stdout:Relocations : (not relocatable) 2024-04-24T22:15:57.005 INFO:teuthology.orchestra.run.smithi080.stdout:Packager : CentOS BuildSystem 2024-04-24T22:15:57.005 INFO:teuthology.orchestra.run.smithi080.stdout:Vendor : CentOS 2024-04-24T22:15:57.005 INFO:teuthology.orchestra.run.smithi080.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-24T22:15:57.005 INFO:teuthology.orchestra.run.smithi080.stdout:Summary : Support tools for GRUB. 2024-04-24T22:15:57.005 INFO:teuthology.orchestra.run.smithi080.stdout:Description : 2024-04-24T22:15:57.005 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:15:57.005 INFO:teuthology.orchestra.run.smithi080.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-24T22:15:57.005 INFO:teuthology.orchestra.run.smithi080.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-24T22:15:57.005 INFO:teuthology.orchestra.run.smithi080.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-24T22:15:57.005 INFO:teuthology.orchestra.run.smithi080.stdout:hardware devices. 2024-04-24T22:15:57.005 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:15:57.005 INFO:teuthology.orchestra.run.smithi080.stdout:This subpackage provides tools for support of all platforms. 2024-04-24T22:15:57.006 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-24T22:15:57.007 INFO:teuthology.task.kernel:Updating grub on smithi080 to boot 4.18.0-552.el8.x86_64 2024-04-24T22:15:57.007 DEBUG:teuthology.orchestra.run.smithi080:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-24T22:15:58.307 INFO:teuthology.orchestra.run.smithi080.stderr:Generating grub configuration file ... 2024-04-24T22:15:59.001 INFO:teuthology.orchestra.run.smithi134.stderr:done 2024-04-24T22:15:59.003 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-04-24T22:15:59.032 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.VRzGESrP1q 2024-04-24T22:15:59.033 DEBUG:teuthology.orchestra.run.smithi134:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.VRzGESrP1q 2024-04-24T22:15:59.120 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.VRzGESrP1q 2024-04-24T22:15:59.219 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.VRzGESrP1q is 6KB 2024-04-24T22:15:59.267 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.VRzGESrP1q 2024-04-24T22:15:59.284 DEBUG:teuthology.orchestra.run.smithi134:> sudo /bin/ls /boot/loader/entries || true 2024-04-24T22:15:59.357 INFO:teuthology.orchestra.run.smithi134.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-24T22:15:59.357 INFO:teuthology.orchestra.run.smithi134.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-24T22:15:59.357 INFO:teuthology.orchestra.run.smithi134.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-24T22:15:59.357 INFO:teuthology.orchestra.run.smithi134.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-24T22:15:59.358 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-24T22:15:59.500 DEBUG:teuthology.orchestra.run.smithi134:> sudo shutdown -r now 2024-04-24T22:16:00.568 INFO:teuthology.orchestra.run.smithi080.stderr:done 2024-04-24T22:16:00.573 DEBUG:teuthology.orchestra.run.smithi080:> mktemp 2024-04-24T22:16:00.603 INFO:teuthology.orchestra.run.smithi080.stdout:/tmp/tmp.P2u0uKXP98 2024-04-24T22:16:00.604 DEBUG:teuthology.orchestra.run.smithi080:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.P2u0uKXP98 2024-04-24T22:16:00.678 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 0666 /tmp/tmp.P2u0uKXP98 2024-04-24T22:16:00.816 DEBUG:teuthology.orchestra.remote:smithi080:/tmp/tmp.P2u0uKXP98 is 6KB 2024-04-24T22:16:00.865 DEBUG:teuthology.orchestra.run.smithi080:> rm -fr /tmp/tmp.P2u0uKXP98 2024-04-24T22:16:00.883 DEBUG:teuthology.orchestra.run.smithi080:> sudo /bin/ls /boot/loader/entries || true 2024-04-24T22:16:00.957 INFO:teuthology.orchestra.run.smithi080.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-24T22:16:00.957 INFO:teuthology.orchestra.run.smithi080.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-24T22:16:00.958 INFO:teuthology.orchestra.run.smithi080.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-24T22:16:00.958 INFO:teuthology.orchestra.run.smithi080.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-24T22:16:00.959 DEBUG:teuthology.orchestra.run.smithi080:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-24T22:16:01.119 DEBUG:teuthology.orchestra.run.smithi080:> sudo shutdown -r now 2024-04-24T22:16:29.530 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-04-24T22:16:29.531 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-24T22:16:29.532 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T22:16:31.122 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-04-24T22:16:31.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-04-24T22:16:31.123 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T22:16:47.902 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2024-04-24T22:16:49.694 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.80 2024-04-24T22:16:56.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-24T22:16:56.912 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T22:16:58.697 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-04-24T22:16:58.698 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T22:16:59.966 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2024-04-24T22:17:01.762 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.80 2024-04-24T22:17:11.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-24T22:17:11.979 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T22:17:13.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-04-24T22:17:13.765 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T22:17:32.190 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.80 2024-04-24T22:17:44.353 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-04-24T22:17:44.801 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-24T22:17:44.801 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "4.18.0-552.el8.x86_64"... 2024-04-24T22:17:44.801 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-04-24T22:17:44.822 INFO:teuthology.orchestra.run.smithi134.stdout:4.18.0-552.el8.x86_64 2024-04-24T22:17:44.822 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-24T22:17:44.822 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-24T22:17:44.822 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-24T22:17:45.823 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-04-24T22:17:45.824 DEBUG:teuthology.orchestra.run.smithi134:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T22:17:45.897 INFO:teuthology.orchestra.run.smithi134.stdout:ttyS1 2024-04-24T22:17:45.912 DEBUG:teuthology.parallel:result is None 2024-04-24T22:17:47.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-04-24T22:17:47.192 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T22:17:48.388 DEBUG:teuthology.orchestra.run.smithi080:> true 2024-04-24T22:17:48.952 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-04-24T22:17:48.952 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "4.18.0-552.el8.x86_64"... 2024-04-24T22:17:48.952 DEBUG:teuthology.orchestra.run.smithi080:> uname -r 2024-04-24T22:17:48.970 INFO:teuthology.orchestra.run.smithi080.stdout:4.18.0-552.el8.x86_64 2024-04-24T22:17:48.971 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-24T22:17:48.971 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-24T22:17:48.971 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-24T22:17:49.972 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-04-24T22:17:49.972 DEBUG:teuthology.orchestra.run.smithi080:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T22:17:50.031 INFO:teuthology.orchestra.run.smithi080.stdout:ttyS1 2024-04-24T22:17:50.048 DEBUG:teuthology.parallel:result is None 2024-04-24T22:17:50.048 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-24T22:17:50.055 INFO:teuthology.task.internal:Creating test directory... 2024-04-24T22:17:50.056 DEBUG:teuthology.orchestra.run.smithi080:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T22:17:50.058 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T22:17:50.080 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-24T22:17:50.087 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-24T22:17:50.144 INFO:teuthology.task.internal:Creating archive directory... 2024-04-24T22:17:50.145 DEBUG:teuthology.orchestra.run.smithi080:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T22:17:50.147 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T22:17:50.179 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-24T22:17:50.186 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-24T22:17:50.186 DEBUG:teuthology.orchestra.run.smithi080:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T22:17:50.213 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T22:17:50.251 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T22:17:50.251 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T22:17:50.262 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T22:17:50.262 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T22:17:50.264 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-24T22:17:50.271 INFO:teuthology.task.internal:Configuring sudo... 2024-04-24T22:17:50.271 DEBUG:teuthology.orchestra.run.smithi080:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T22:17:50.305 DEBUG:teuthology.orchestra.run.smithi134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T22:17:50.335 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-24T22:17:50.345 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-24T22:17:50.345 DEBUG:teuthology.orchestra.run.smithi080:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T22:17:50.375 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T22:17:50.394 DEBUG:teuthology.orchestra.run.smithi080:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T22:17:50.474 DEBUG:teuthology.orchestra.run.smithi080:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T22:17:50.521 DEBUG:teuthology.orchestra.run.smithi080:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T22:17:50.600 DEBUG:teuthology.orchestra.run.smithi080:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T22:17:50.643 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:17:50.643 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T22:17:50.709 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T22:17:50.752 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T22:17:50.805 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T22:17:50.891 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T22:17:50.935 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:17:50.935 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T22:17:51.003 DEBUG:teuthology.orchestra.run.smithi080:> sudo service rsyslog restart 2024-04-24T22:17:51.005 DEBUG:teuthology.orchestra.run.smithi134:> sudo service rsyslog restart 2024-04-24T22:17:51.061 INFO:teuthology.orchestra.run.smithi080.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T22:17:51.091 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T22:17:51.579 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-24T22:17:51.587 INFO:teuthology.task.internal:Starting timer... 2024-04-24T22:17:51.587 INFO:teuthology.run_tasks:Running task pcp... 2024-04-24T22:17:51.645 INFO:teuthology.run_tasks:Running task selinux... 2024-04-24T22:17:51.653 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-24T22:17:51.654 DEBUG:teuthology.orchestra.run.smithi080:> sudo service auditd rotate 2024-04-24T22:17:51.723 INFO:teuthology.orchestra.run.smithi080.stdout:Rotating logs: 2024-04-24T22:17:51.724 DEBUG:teuthology.orchestra.run.smithi134:> sudo service auditd rotate 2024-04-24T22:17:51.821 INFO:teuthology.orchestra.run.smithi134.stdout:Rotating logs: 2024-04-24T22:17:51.823 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-24T22:17:51.823 DEBUG:teuthology.orchestra.run.smithi080:> /usr/sbin/getenforce 2024-04-24T22:17:51.848 INFO:teuthology.orchestra.run.smithi080.stdout:Permissive 2024-04-24T22:17:51.849 DEBUG:teuthology.orchestra.run.smithi134:> /usr/sbin/getenforce 2024-04-24T22:17:51.874 INFO:teuthology.orchestra.run.smithi134.stdout:Permissive 2024-04-24T22:17:51.874 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi080.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi134.front.sepia.ceph.com': 'permissive'} 2024-04-24T22:17:51.874 DEBUG:teuthology.orchestra.run.smithi080:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-24T22:17:51.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:17:51.918 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-24T22:17:51.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:17:51.945 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-24T22:17:51.945 DEBUG:teuthology.orchestra.run.smithi080:> sudo /usr/sbin/setenforce permissive 2024-04-24T22:17:51.985 DEBUG:teuthology.orchestra.run.smithi134:> sudo /usr/sbin/setenforce permissive 2024-04-24T22:17:52.012 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-24T22:17:52.021 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-24T22:17:52.027 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-24T22:17:52.154 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-24T22:17:52.244 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-24T22:17:52.336 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi080.front.sepia.ceph.com,smithi134.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-24T22:23:46.209 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi080.front.sepia.ceph.com'), Remote(name='ubuntu@smithi134.front.sepia.ceph.com')] 2024-04-24T22:23:46.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-04-24T22:23:46.228 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T22:23:46.306 DEBUG:teuthology.orchestra.run.smithi080:> true 2024-04-24T22:23:46.374 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-04-24T22:23:46.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-24T22:23:46.375 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T22:23:46.453 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-04-24T22:23:46.481 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-24T22:23:46.482 INFO:teuthology.run_tasks:Running task clock... 2024-04-24T22:23:46.492 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-24T22:23:46.493 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T22:23:46.493 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T22:23:46.495 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T22:23:46.496 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T22:23:46.525 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T22:23:46.539 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T22:23:46.555 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T22:23:46.569 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T22:23:46.602 INFO:teuthology.orchestra.run.smithi080.stderr:sudo: ntpd: command not found 2024-04-24T22:23:46.611 INFO:teuthology.orchestra.run.smithi080.stdout:506 Cannot talk to daemon 2024-04-24T22:23:46.624 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T22:23:46.637 INFO:teuthology.orchestra.run.smithi080.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T22:23:46.641 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: ntpd: command not found 2024-04-24T22:23:46.652 INFO:teuthology.orchestra.run.smithi134.stdout:506 Cannot talk to daemon 2024-04-24T22:23:46.665 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T22:23:46.678 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T22:23:46.700 INFO:teuthology.orchestra.run.smithi080.stderr:bash: ntpq: command not found 2024-04-24T22:23:46.703 INFO:teuthology.orchestra.run.smithi080.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T22:23:46.704 INFO:teuthology.orchestra.run.smithi080.stdout:=============================================================================== 2024-04-24T22:23:46.704 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T22:23:46.704 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T22:23:46.704 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T22:23:46.704 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T22:23:46.741 INFO:teuthology.orchestra.run.smithi134.stderr:bash: ntpq: command not found 2024-04-24T22:23:46.744 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T22:23:46.744 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-04-24T22:23:46.744 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T22:23:46.744 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T22:23:46.744 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T22:23:46.744 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T22:23:46.745 INFO:teuthology.run_tasks:Running task install... 2024-04-24T22:23:46.755 DEBUG:teuthology.task.install:project ceph 2024-04-24T22:23:46.755 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}} 2024-04-24T22:23:46.755 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'} 2024-04-24T22:23:46.755 INFO:teuthology.task.install:Using flavor: default 2024-04-24T22:23:46.761 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-24T22:23:46.761 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-04-24T22:23:46.761 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9', 'tag': None, 'wait_for_package': False} 2024-04-24T22:23:46.762 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T22:23:46.763 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9', 'tag': None, 'wait_for_package': False} 2024-04-24T22:23:46.764 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T22:23:46.905 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/ 2024-04-24T22:23:46.905 INFO:teuthology.task.install.rpm:Package version is 17.2.7-915.g330067db 2024-04-24T22:23:47.056 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/ 2024-04-24T22:23:47.056 INFO:teuthology.task.install.rpm:Package version is 17.2.7-915.g330067db 2024-04-24T22:23:47.111 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T22:23:47.112 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:23:47.112 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T22:23:47.141 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-04-24T22:23:47.142 DEBUG:teuthology.orchestra.run.smithi080:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/330067dbce47655d1f762ef14d546d2fde16a6a9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T22:23:47.166 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T22:23:47.167 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:23:47.167 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T22:23:47.196 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-04-24T22:23:47.196 DEBUG:teuthology.orchestra.run.smithi134:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/330067dbce47655d1f762ef14d546d2fde16a6a9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T22:23:47.210 DEBUG:teuthology.orchestra.run.smithi080:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T22:23:47.265 DEBUG:teuthology.orchestra.run.smithi134:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T22:23:47.289 DEBUG:teuthology.orchestra.run.smithi080:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T22:23:47.342 DEBUG:teuthology.orchestra.run.smithi134:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T22:23:47.356 INFO:teuthology.orchestra.run.smithi080.stdout:check_obsoletes = 1 2024-04-24T22:23:47.357 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum clean all 2024-04-24T22:23:47.408 INFO:teuthology.orchestra.run.smithi134.stdout:check_obsoletes = 1 2024-04-24T22:23:47.410 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum clean all 2024-04-24T22:23:47.624 INFO:teuthology.orchestra.run.smithi080.stdout:78 files removed 2024-04-24T22:23:47.655 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-radosgw 2024-04-24T22:23:47.676 INFO:teuthology.orchestra.run.smithi134.stdout:78 files removed 2024-04-24T22:23:47.704 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-radosgw 2024-04-24T22:23:48.461 INFO:teuthology.orchestra.run.smithi134.stdout:ceph packages for x86_64 223 kB/s | 78 kB 00:00 2024-04-24T22:23:48.548 INFO:teuthology.orchestra.run.smithi080.stdout:ceph packages for x86_64 160 kB/s | 78 kB 00:00 2024-04-24T22:23:48.752 INFO:teuthology.orchestra.run.smithi134.stdout:ceph noarch packages 42 kB/s | 11 kB 00:00 2024-04-24T22:23:48.885 INFO:teuthology.orchestra.run.smithi080.stdout:ceph noarch packages 36 kB/s | 11 kB 00:00 2024-04-24T22:23:49.012 INFO:teuthology.orchestra.run.smithi134.stdout:ceph source packages 6.3 kB/s | 1.4 kB 00:00 2024-04-24T22:23:49.336 INFO:teuthology.orchestra.run.smithi080.stdout:ceph source packages 3.4 kB/s | 1.4 kB 00:00 2024-04-24T22:23:49.998 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - AppStream 29 MB/s | 28 MB 00:00 2024-04-24T22:23:50.327 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 8 - AppStream 29 MB/s | 28 MB 00:00 2024-04-24T22:23:55.979 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - BaseOS 11 MB/s | 10 MB 00:00 2024-04-24T22:23:56.236 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 8 - BaseOS 11 MB/s | 10 MB 00:00 2024-04-24T22:23:57.796 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - Extras 83 kB/s | 18 kB 00:00 2024-04-24T22:23:58.044 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 8 - Extras 87 kB/s | 18 kB 00:00 2024-04-24T22:23:58.288 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 7.7 kB 00:00 2024-04-24T22:23:58.570 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 7.7 kB 00:00 2024-04-24T22:23:58.731 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 8 - PowerTools 14 MB/s | 5.0 MB 00:00 2024-04-24T22:23:59.154 INFO:teuthology.orchestra.run.smithi080.stdout:CentOS Stream 8 - PowerTools 10 MB/s | 5.0 MB 00:00 2024-04-24T22:23:59.791 INFO:teuthology.orchestra.run.smithi134.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-04-24T22:24:00.197 INFO:teuthology.orchestra.run.smithi080.stdout:Copr repo for python3-asyncssh owned by ceph 30 kB/s | 3.5 kB 00:00 2024-04-24T22:24:01.303 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2024-04-24T22:24:05.192 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 439 kB/s | 24 kB 00:00 2024-04-24T22:24:08.830 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:24:08.833 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:24:08.833 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-04-24T22:24:08.833 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:24:08.833 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-24T22:24:08.833 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw x86_64 2:17.2.7-915.g330067db.el8 ceph 17 M 2024-04-24T22:24:08.833 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrading: 2024-04-24T22:24:08.833 INFO:teuthology.orchestra.run.smithi134.stdout: librados2 x86_64 2:17.2.7-915.g330067db.el8 ceph 3.5 M 2024-04-24T22:24:08.833 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1 x86_64 2:17.2.7-915.g330067db.el8 ceph 3.6 M 2024-04-24T22:24:08.833 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-04-24T22:24:08.833 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base x86_64 2:17.2.7-915.g330067db.el8 ceph 5.3 M 2024-04-24T22:24:08.833 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common x86_64 2:17.2.7-915.g330067db.el8 ceph 24 M 2024-04-24T22:24:08.833 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux x86_64 2:17.2.7-915.g330067db.el8 ceph 24 k 2024-04-24T22:24:08.833 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-24T22:24:08.834 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2 x86_64 2:17.2.7-915.g330067db.el8 ceph 798 k 2024-04-24T22:24:08.834 INFO:teuthology.orchestra.run.smithi134.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-24T22:24:08.834 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-24T22:24:08.834 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1 x86_64 2:17.2.7-915.g330067db.el8 ceph 505 k 2024-04-24T22:24:08.834 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-24T22:24:08.834 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2 x86_64 2:17.2.7-915.g330067db.el8 ceph 5.6 M 2024-04-24T22:24:08.834 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-24T22:24:08.834 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-24T22:24:08.834 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse x86_64 2:17.2.7-915.g330067db.el8 ceph 45 k 2024-04-24T22:24:08.834 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common x86_64 2:17.2.7-915.g330067db.el8 ceph 117 k 2024-04-24T22:24:08.834 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs x86_64 2:17.2.7-915.g330067db.el8 ceph 190 k 2024-04-24T22:24:08.834 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-24T22:24:08.834 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados x86_64 2:17.2.7-915.g330067db.el8 ceph 370 k 2024-04-24T22:24:08.835 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd x86_64 2:17.2.7-915.g330067db.el8 ceph 358 k 2024-04-24T22:24:08.835 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw x86_64 2:17.2.7-915.g330067db.el8 ceph 107 k 2024-04-24T22:24:08.835 INFO:teuthology.orchestra.run.smithi134.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-24T22:24:08.835 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:24:08.835 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-24T22:24:08.835 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:24:08.835 INFO:teuthology.orchestra.run.smithi134.stdout:Install 21 Packages 2024-04-24T22:24:08.835 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrade 2 Packages 2024-04-24T22:24:08.835 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:24:08.836 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 64 M 2024-04-24T22:24:08.836 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-24T22:24:09.952 INFO:teuthology.orchestra.run.smithi134.stdout:(1/23): ceph-base-17.2.7-915.g330067db.el8.x86_ 8.0 MB/s | 5.3 MB 00:00 2024-04-24T22:24:10.010 INFO:teuthology.orchestra.run.smithi134.stdout:(2/23): ceph-selinux-17.2.7-915.g330067db.el8.x 419 kB/s | 24 kB 00:00 2024-04-24T22:24:10.141 INFO:teuthology.orchestra.run.smithi134.stdout:(3/23): libcephfs2-17.2.7-915.g330067db.el8.x86 6.0 MB/s | 798 kB 00:00 2024-04-24T22:24:10.228 INFO:teuthology.orchestra.run.smithi134.stdout:(4/23): libradosstriper1-17.2.7-915.g330067db.e 5.7 MB/s | 505 kB 00:00 2024-04-24T22:24:10.737 INFO:teuthology.orchestra.run.smithi134.stdout:(5/23): librgw2-17.2.7-915.g330067db.el8.x86_64 11 MB/s | 5.6 MB 00:00 2024-04-24T22:24:10.804 INFO:teuthology.orchestra.run.smithi134.stdout:(6/23): python3-ceph-argparse-17.2.7-915.g33006 674 kB/s | 45 kB 00:00 2024-04-24T22:24:10.863 INFO:teuthology.orchestra.run.smithi134.stdout:(7/23): python3-ceph-common-17.2.7-915.g330067d 2.0 MB/s | 117 kB 00:00 2024-04-24T22:24:10.937 INFO:teuthology.orchestra.run.smithi134.stdout:(8/23): python3-cephfs-17.2.7-915.g330067db.el8 2.5 MB/s | 190 kB 00:00 2024-04-24T22:24:11.165 INFO:teuthology.orchestra.run.smithi134.stdout:(9/23): ceph-common-17.2.7-915.g330067db.el8.x8 13 MB/s | 24 MB 00:01 2024-04-24T22:24:11.188 INFO:teuthology.orchestra.run.smithi134.stdout:(10/23): python3-rados-17.2.7-915.g330067db.el8 1.4 MB/s | 370 kB 00:00 2024-04-24T22:24:11.223 INFO:teuthology.orchestra.run.smithi134.stdout:(11/23): python3-rbd-17.2.7-915.g330067db.el8.x 6.0 MB/s | 358 kB 00:00 2024-04-24T22:24:11.247 INFO:teuthology.orchestra.run.smithi134.stdout:(12/23): python3-rgw-17.2.7-915.g330067db.el8.x 1.8 MB/s | 107 kB 00:00 2024-04-24T22:24:11.473 INFO:teuthology.orchestra.run.smithi134.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.1 MB/s | 259 kB 00:00 2024-04-24T22:24:11.516 INFO:teuthology.orchestra.run.smithi134.stdout:(14/23): librdkafka-1.6.1-1.el8.x86_64.rpm 2.3 MB/s | 677 kB 00:00 2024-04-24T22:24:11.539 INFO:teuthology.orchestra.run.smithi134.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 657 kB/s | 44 kB 00:00 2024-04-24T22:24:11.748 INFO:teuthology.orchestra.run.smithi134.stdout:(16/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 201 kB/s | 47 kB 00:00 2024-04-24T22:24:11.782 INFO:teuthology.orchestra.run.smithi134.stdout:(17/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2024-04-24T22:24:11.857 INFO:teuthology.orchestra.run.smithi134.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.0 MB/s | 75 kB 00:00 2024-04-24T22:24:11.924 INFO:teuthology.orchestra.run.smithi134.stdout:(19/23): liboath-2.6.2-3.el8.x86_64.rpm 336 kB/s | 59 kB 00:00 2024-04-24T22:24:12.132 INFO:teuthology.orchestra.run.smithi134.stdout:(20/23): ceph-radosgw-17.2.7-915.g330067db.el8. 5.9 MB/s | 17 MB 00:02 2024-04-24T22:24:12.174 INFO:teuthology.orchestra.run.smithi134.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 5.5 MB/s | 1.7 MB 00:00 2024-04-24T22:24:12.266 INFO:teuthology.orchestra.run.smithi134.stdout:(22/23): librados2-17.2.7-915.g330067db.el8.x86 10 MB/s | 3.5 MB 00:00 2024-04-24T22:24:12.467 INFO:teuthology.orchestra.run.smithi134.stdout:(23/23): librbd1-17.2.7-915.g330067db.el8.x86_6 11 MB/s | 3.6 MB 00:00 2024-04-24T22:24:12.469 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:24:12.469 INFO:teuthology.orchestra.run.smithi134.stdout:Total 18 MB/s | 64 MB 00:03 2024-04-24T22:24:12.513 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-24T22:24:12.559 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-24T22:24:12.559 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-24T22:24:12.911 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-24T22:24:12.914 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-24T22:24:13.929 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-24T22:24:14.145 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-04-24T22:24:14.326 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-24T22:24:14.705 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-24T22:24:14.934 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librados2-2:17.2.7-915.g330067db.el8.x86_64 3/25 2024-04-24T22:24:15.032 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:17.2.7-915.g330067db.el8.x86_64 3/25 2024-04-24T22:24:15.276 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rados-2:17.2.7-915.g330067db.el8.x86_64 4/25 2024-04-24T22:24:15.468 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 5/25 2024-04-24T22:24:15.841 INFO:teuthology.orchestra.run.smithi080.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 16 MB 00:15 2024-04-24T22:24:15.883 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 5/25 2024-04-24T22:24:16.067 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librbd1-2:17.2.7-915.g330067db.el8.x86_64 6/25 2024-04-24T22:24:16.284 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:17.2.7-915.g330067db.el8.x86_64 6/25 2024-04-24T22:24:16.505 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-04-24T22:24:16.735 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-04-24T22:24:17.429 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-04-24T22:24:17.592 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librgw2-2:17.2.7-915.g330067db.el8.x86_64 10/25 2024-04-24T22:24:17.658 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librgw2-2:17.2.7-915.g330067db.el8.x86_64 10/25 2024-04-24T22:24:17.829 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-argparse-2:17.2.7-915.g330067db.el8.x 11/25 2024-04-24T22:24:17.997 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 12/25 2024-04-24T22:24:18.202 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 13/25 2024-04-24T22:24:18.437 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 14/25 2024-04-24T22:24:18.609 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libradosstriper1-2:17.2.7-915.g330067db.el8.x86_64 15/25 2024-04-24T22:24:18.681 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libradosstriper1-2:17.2.7-915.g330067db.el8.x86_64 15/25 2024-04-24T22:24:18.885 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-24T22:24:19.084 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-04-24T22:24:19.293 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-04-24T22:24:19.509 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-common-2:17.2.7-915.g330067db.el8.x86 19/25 2024-04-24T22:24:19.713 INFO:teuthology.orchestra.run.smithi080.stdout:lab-extras 331 kB/s | 24 kB 00:00 2024-04-24T22:24:22.163 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:17.2.7-915.g330067db.el8.x86_64 20/25 2024-04-24T22:24:22.350 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-common-2:17.2.7-915.g330067db.el8.x86_64 20/25 2024-04-24T22:24:22.934 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:17.2.7-915.g330067db.el8.x86_64 20/25 2024-04-24T22:24:23.224 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-base-2:17.2.7-915.g330067db.el8.x86_64 21/25 2024-04-24T22:24:23.314 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-base-2:17.2.7-915.g330067db.el8.x86_64 21/25 2024-04-24T22:24:23.359 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:24:23.361 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:24:23.362 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-04-24T22:24:23.362 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:24:23.362 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-04-24T22:24:23.362 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-radosgw x86_64 2:17.2.7-915.g330067db.el8 ceph 17 M 2024-04-24T22:24:23.362 INFO:teuthology.orchestra.run.smithi080.stdout:Upgrading: 2024-04-24T22:24:23.362 INFO:teuthology.orchestra.run.smithi080.stdout: librados2 x86_64 2:17.2.7-915.g330067db.el8 ceph 3.5 M 2024-04-24T22:24:23.362 INFO:teuthology.orchestra.run.smithi080.stdout: librbd1 x86_64 2:17.2.7-915.g330067db.el8 ceph 3.6 M 2024-04-24T22:24:23.362 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-04-24T22:24:23.362 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-base x86_64 2:17.2.7-915.g330067db.el8 ceph 5.3 M 2024-04-24T22:24:23.362 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-common x86_64 2:17.2.7-915.g330067db.el8 ceph 24 M 2024-04-24T22:24:23.362 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-selinux x86_64 2:17.2.7-915.g330067db.el8 ceph 24 k 2024-04-24T22:24:23.362 INFO:teuthology.orchestra.run.smithi080.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-24T22:24:23.362 INFO:teuthology.orchestra.run.smithi080.stdout: libcephfs2 x86_64 2:17.2.7-915.g330067db.el8 ceph 798 k 2024-04-24T22:24:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-24T22:24:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-24T22:24:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: libradosstriper1 x86_64 2:17.2.7-915.g330067db.el8 ceph 505 k 2024-04-24T22:24:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-24T22:24:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: librgw2 x86_64 2:17.2.7-915.g330067db.el8 ceph 5.6 M 2024-04-24T22:24:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-24T22:24:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-24T22:24:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ceph-argparse x86_64 2:17.2.7-915.g330067db.el8 ceph 45 k 2024-04-24T22:24:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ceph-common x86_64 2:17.2.7-915.g330067db.el8 ceph 117 k 2024-04-24T22:24:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cephfs x86_64 2:17.2.7-915.g330067db.el8 ceph 190 k 2024-04-24T22:24:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-24T22:24:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rados x86_64 2:17.2.7-915.g330067db.el8 ceph 370 k 2024-04-24T22:24:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rbd x86_64 2:17.2.7-915.g330067db.el8 ceph 358 k 2024-04-24T22:24:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rgw x86_64 2:17.2.7-915.g330067db.el8 ceph 107 k 2024-04-24T22:24:23.363 INFO:teuthology.orchestra.run.smithi080.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-24T22:24:23.364 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:24:23.364 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-04-24T22:24:23.364 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:24:23.364 INFO:teuthology.orchestra.run.smithi080.stdout:Install 21 Packages 2024-04-24T22:24:23.364 INFO:teuthology.orchestra.run.smithi080.stdout:Upgrade 2 Packages 2024-04-24T22:24:23.364 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:24:23.364 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 64 M 2024-04-24T22:24:23.364 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-04-24T22:24:23.458 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 22/25 2024-04-24T22:24:24.539 INFO:teuthology.orchestra.run.smithi080.stdout:(1/23): ceph-base-17.2.7-915.g330067db.el8.x86_ 6.6 MB/s | 5.3 MB 00:00 2024-04-24T22:24:24.798 INFO:teuthology.orchestra.run.smithi080.stdout:(2/23): ceph-selinux-17.2.7-915.g330067db.el8.x 325 kB/s | 24 kB 00:00 2024-04-24T22:24:25.381 INFO:teuthology.orchestra.run.smithi080.stdout:(3/23): ceph-common-17.2.7-915.g330067db.el8.x8 14 MB/s | 24 MB 00:01 2024-04-24T22:24:25.415 INFO:teuthology.orchestra.run.smithi080.stdout:(4/23): libcephfs2-17.2.7-915.g330067db.el8.x86 997 kB/s | 798 kB 00:00 2024-04-24T22:24:25.449 INFO:teuthology.orchestra.run.smithi080.stdout:(5/23): libradosstriper1-17.2.7-915.g330067db.e 7.4 MB/s | 505 kB 00:00 2024-04-24T22:24:25.507 INFO:teuthology.orchestra.run.smithi080.stdout:(6/23): python3-ceph-argparse-17.2.7-915.g33006 768 kB/s | 45 kB 00:00 2024-04-24T22:24:25.574 INFO:teuthology.orchestra.run.smithi080.stdout:(7/23): python3-ceph-common-17.2.7-915.g330067d 1.7 MB/s | 117 kB 00:00 2024-04-24T22:24:25.633 INFO:teuthology.orchestra.run.smithi080.stdout:(8/23): python3-cephfs-17.2.7-915.g330067db.el8 3.2 MB/s | 190 kB 00:00 2024-04-24T22:24:25.708 INFO:teuthology.orchestra.run.smithi080.stdout:(9/23): python3-rados-17.2.7-915.g330067db.el8. 4.8 MB/s | 370 kB 00:00 2024-04-24T22:24:25.808 INFO:teuthology.orchestra.run.smithi080.stdout:(10/23): librgw2-17.2.7-915.g330067db.el8.x86_6 14 MB/s | 5.6 MB 00:00 2024-04-24T22:24:25.833 INFO:teuthology.orchestra.run.smithi080.stdout:(11/23): python3-rbd-17.2.7-915.g330067db.el8.x 2.8 MB/s | 358 kB 00:00 2024-04-24T22:24:25.858 INFO:teuthology.orchestra.run.smithi080.stdout:(12/23): python3-rgw-17.2.7-915.g330067db.el8.x 2.1 MB/s | 107 kB 00:00 2024-04-24T22:24:26.076 INFO:teuthology.orchestra.run.smithi080.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.2 MB/s | 259 kB 00:00 2024-04-24T22:24:26.109 INFO:teuthology.orchestra.run.smithi080.stdout:(14/23): librdkafka-1.6.1-1.el8.x86_64.rpm 2.4 MB/s | 677 kB 00:00 2024-04-24T22:24:26.135 INFO:teuthology.orchestra.run.smithi080.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 748 kB/s | 44 kB 00:00 2024-04-24T22:24:26.343 INFO:teuthology.orchestra.run.smithi080.stdout:(16/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 201 kB/s | 47 kB 00:00 2024-04-24T22:24:26.652 INFO:teuthology.orchestra.run.smithi080.stdout:(17/23): gperftools-libs-2.7-9.el8.x86_64.rpm 592 kB/s | 306 kB 00:00 2024-04-24T22:24:26.694 INFO:teuthology.orchestra.run.smithi080.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 168 kB/s | 59 kB 00:00 2024-04-24T22:24:26.778 INFO:teuthology.orchestra.run.smithi080.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 600 kB/s | 75 kB 00:00 2024-04-24T22:24:26.995 INFO:teuthology.orchestra.run.smithi080.stdout:(20/23): ceph-radosgw-17.2.7-915.g330067db.el8. 5.2 MB/s | 17 MB 00:03 2024-04-24T22:24:27.155 INFO:teuthology.orchestra.run.smithi080.stdout:(21/23): librados2-17.2.7-915.g330067db.el8.x86 9.2 MB/s | 3.5 MB 00:00 2024-04-24T22:24:27.347 INFO:teuthology.orchestra.run.smithi080.stdout:(22/23): librbd1-17.2.7-915.g330067db.el8.x86_6 10 MB/s | 3.6 MB 00:00 2024-04-24T22:24:27.387 INFO:teuthology.orchestra.run.smithi080.stdout:(23/23): thrift-0.13.0-2.el8.x86_64.rpm 2.5 MB/s | 1.7 MB 00:00 2024-04-24T22:24:27.391 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:24:27.391 INFO:teuthology.orchestra.run.smithi080.stdout:Total 16 MB/s | 64 MB 00:04 2024-04-24T22:24:27.438 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-04-24T22:24:27.477 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-04-24T22:24:27.481 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-04-24T22:24:27.849 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-04-24T22:24:27.852 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-04-24T22:24:28.877 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-04-24T22:24:29.126 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-04-24T22:24:29.322 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-24T22:24:29.710 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-24T22:24:29.947 INFO:teuthology.orchestra.run.smithi080.stdout: Upgrading : librados2-2:17.2.7-915.g330067db.el8.x86_64 3/25 2024-04-24T22:24:30.044 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librados2-2:17.2.7-915.g330067db.el8.x86_64 3/25 2024-04-24T22:24:30.329 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-rados-2:17.2.7-915.g330067db.el8.x86_64 4/25 2024-04-24T22:24:30.522 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 5/25 2024-04-24T22:24:30.913 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 5/25 2024-04-24T22:24:31.130 INFO:teuthology.orchestra.run.smithi080.stdout: Upgrading : librbd1-2:17.2.7-915.g330067db.el8.x86_64 6/25 2024-04-24T22:24:31.341 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librbd1-2:17.2.7-915.g330067db.el8.x86_64 6/25 2024-04-24T22:24:31.597 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-04-24T22:24:31.826 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-04-24T22:24:32.486 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-04-24T22:24:32.646 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : librgw2-2:17.2.7-915.g330067db.el8.x86_64 10/25 2024-04-24T22:24:32.713 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librgw2-2:17.2.7-915.g330067db.el8.x86_64 10/25 2024-04-24T22:24:32.890 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-ceph-argparse-2:17.2.7-915.g330067db.el8.x 11/25 2024-04-24T22:24:33.064 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 12/25 2024-04-24T22:24:33.306 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 13/25 2024-04-24T22:24:33.544 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 14/25 2024-04-24T22:24:33.704 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libradosstriper1-2:17.2.7-915.g330067db.el8.x86_64 15/25 2024-04-24T22:24:33.761 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libradosstriper1-2:17.2.7-915.g330067db.el8.x86_64 15/25 2024-04-24T22:24:33.964 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-24T22:24:34.205 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-04-24T22:24:34.398 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-04-24T22:24:34.604 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-ceph-common-2:17.2.7-915.g330067db.el8.x86 19/25 2024-04-24T22:24:37.283 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-common-2:17.2.7-915.g330067db.el8.x86_64 20/25 2024-04-24T22:24:37.512 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-common-2:17.2.7-915.g330067db.el8.x86_64 20/25 2024-04-24T22:24:38.084 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-common-2:17.2.7-915.g330067db.el8.x86_64 20/25 2024-04-24T22:24:38.385 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-base-2:17.2.7-915.g330067db.el8.x86_64 21/25 2024-04-24T22:24:38.472 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-base-2:17.2.7-915.g330067db.el8.x86_64 21/25 2024-04-24T22:24:38.612 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 22/25 2024-04-24T22:24:42.888 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 22/25 2024-04-24T22:24:42.888 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /sys 2024-04-24T22:24:42.888 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /proc 2024-04-24T22:24:42.888 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /dev 2024-04-24T22:24:42.888 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /run 2024-04-24T22:24:42.888 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /mnt 2024-04-24T22:24:42.888 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /var/tmp 2024-04-24T22:24:42.888 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /home 2024-04-24T22:24:42.888 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /tmp 2024-04-24T22:24:42.889 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /dev 2024-04-24T22:24:42.889 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:24:44.571 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 23/25 2024-04-24T22:24:44.635 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 23/25 2024-04-24T22:24:44.636 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-24T22:24:44.848 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-24T22:24:44.849 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-24T22:24:45.554 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-24T22:24:45.555 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-base-2:17.2.7-915.g330067db.el8.x86_64 1/25 2024-04-24T22:24:45.555 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-common-2:17.2.7-915.g330067db.el8.x86_64 2/25 2024-04-24T22:24:45.555 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 3/25 2024-04-24T22:24:45.555 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 4/25 2024-04-24T22:24:45.555 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 5/25 2024-04-24T22:24:45.555 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libradosstriper1-2:17.2.7-915.g330067db.el8.x86_64 6/25 2024-04-24T22:24:45.555 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librgw2-2:17.2.7-915.g330067db.el8.x86_64 7/25 2024-04-24T22:24:45.555 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-argparse-2:17.2.7-915.g330067db.el8.x 8/25 2024-04-24T22:24:45.555 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-common-2:17.2.7-915.g330067db.el8.x86 9/25 2024-04-24T22:24:45.555 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 10/25 2024-04-24T22:24:45.555 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rados-2:17.2.7-915.g330067db.el8.x86_64 11/25 2024-04-24T22:24:45.555 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 12/25 2024-04-24T22:24:45.555 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 13/25 2024-04-24T22:24:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-04-24T22:24:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-04-24T22:24:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-04-24T22:24:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-04-24T22:24:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-04-24T22:24:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-04-24T22:24:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-04-24T22:24:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-04-24T22:24:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:17.2.7-915.g330067db.el8.x86_64 22/25 2024-04-24T22:24:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-24T22:24:45.556 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:17.2.7-915.g330067db.el8.x86_64 24/25 2024-04-24T22:24:46.447 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-24T22:24:46.448 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:24:46.448 INFO:teuthology.orchestra.run.smithi134.stdout:Upgraded: 2024-04-24T22:24:46.448 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:24:46.448 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:24:46.448 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-24T22:24:46.448 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:24:46.448 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:24:46.448 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:24:46.448 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:24:46.448 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-24T22:24:46.448 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:24:46.449 INFO:teuthology.orchestra.run.smithi134.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-24T22:24:46.449 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-24T22:24:46.449 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:24:46.449 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-24T22:24:46.449 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:24:46.449 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-24T22:24:46.449 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-24T22:24:46.449 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:24:46.449 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:24:46.449 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:24:46.449 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-24T22:24:46.449 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:24:46.449 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:24:46.450 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:24:46.450 INFO:teuthology.orchestra.run.smithi134.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-24T22:24:46.450 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:24:46.450 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:24:46.765 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-test 2024-04-24T22:24:47.505 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:42 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:24:49.170 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:24:49.171 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:24:49.171 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-04-24T22:24:49.171 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:24:49.171 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-24T22:24:49.172 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test x86_64 2:17.2.7-915.g330067db.el8 ceph 53 M 2024-04-24T22:24:49.172 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-04-24T22:24:49.172 INFO:teuthology.orchestra.run.smithi134.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-24T22:24:49.172 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite x86_64 2:17.2.7-915.g330067db.el8 ceph 179 k 2024-04-24T22:24:49.172 INFO:teuthology.orchestra.run.smithi134.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-24T22:24:49.172 INFO:teuthology.orchestra.run.smithi134.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-24T22:24:49.172 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-24T22:24:49.172 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:24:49.172 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-24T22:24:49.172 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:24:49.173 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2024-04-24T22:24:49.173 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:24:49.173 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 54 M 2024-04-24T22:24:49.173 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 236 M 2024-04-24T22:24:49.173 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-24T22:24:49.435 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): libcephsqlite-17.2.7-915.g330067db.el8.x 894 kB/s | 179 kB 00:00 2024-04-24T22:24:49.460 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 898 kB/s | 203 kB 00:00 2024-04-24T22:24:49.610 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.1 MB/s | 323 kB 00:00 2024-04-24T22:24:49.652 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 867 kB/s | 188 kB 00:00 2024-04-24T22:24:49.702 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 769 kB/s | 71 kB 00:00 2024-04-24T22:24:51.952 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): ceph-test-17.2.7-915.g330067db.el8.x86_6 19 MB/s | 53 MB 00:02 2024-04-24T22:24:51.954 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:24:51.955 INFO:teuthology.orchestra.run.smithi134.stdout:Total 19 MB/s | 54 MB 00:02 2024-04-24T22:24:51.997 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-24T22:24:52.018 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-24T22:24:52.018 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-24T22:24:52.347 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-24T22:24:52.350 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-24T22:24:52.818 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-24T22:24:52.998 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-04-24T22:24:53.192 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-04-24T22:24:53.328 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-24T22:24:53.390 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-24T22:24:53.602 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-04-24T22:24:53.736 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 5/6 2024-04-24T22:24:57.818 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 22/25 2024-04-24T22:24:57.818 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /sys 2024-04-24T22:24:57.818 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /proc 2024-04-24T22:24:57.818 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /dev 2024-04-24T22:24:57.818 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /run 2024-04-24T22:24:57.819 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /mnt 2024-04-24T22:24:57.819 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /var/tmp 2024-04-24T22:24:57.819 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /home 2024-04-24T22:24:57.819 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /tmp 2024-04-24T22:24:57.819 INFO:teuthology.orchestra.run.smithi080.stdout:skipping the directory /dev 2024-04-24T22:24:57.819 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:24:58.774 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 5/6 2024-04-24T22:24:59.018 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-test-2:17.2.7-915.g330067db.el8.x86_64 6/6 2024-04-24T22:24:59.290 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-test-2:17.2.7-915.g330067db.el8.x86_64 6/6 2024-04-24T22:24:59.292 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-test-2:17.2.7-915.g330067db.el8.x86_64 1/6 2024-04-24T22:24:59.292 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 2/6 2024-04-24T22:24:59.292 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-04-24T22:24:59.292 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-04-24T22:24:59.292 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-04-24T22:24:59.523 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 23/25 2024-04-24T22:24:59.597 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 23/25 2024-04-24T22:24:59.597 INFO:teuthology.orchestra.run.smithi080.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-24T22:24:59.727 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-04-24T22:24:59.727 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:24:59.727 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-24T22:24:59.727 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:24:59.728 INFO:teuthology.orchestra.run.smithi134.stdout: jq-1.6-8.el8.x86_64 2024-04-24T22:24:59.728 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:24:59.728 INFO:teuthology.orchestra.run.smithi134.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-04-24T22:24:59.728 INFO:teuthology.orchestra.run.smithi134.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-24T22:24:59.728 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-24T22:24:59.728 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:24:59.728 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:24:59.826 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-24T22:24:59.826 INFO:teuthology.orchestra.run.smithi080.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-24T22:24:59.972 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph 2024-04-24T22:25:00.531 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-24T22:25:00.532 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-base-2:17.2.7-915.g330067db.el8.x86_64 1/25 2024-04-24T22:25:00.532 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-common-2:17.2.7-915.g330067db.el8.x86_64 2/25 2024-04-24T22:25:00.532 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 3/25 2024-04-24T22:25:00.532 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 4/25 2024-04-24T22:25:00.532 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 5/25 2024-04-24T22:25:00.532 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libradosstriper1-2:17.2.7-915.g330067db.el8.x86_64 6/25 2024-04-24T22:25:00.532 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librgw2-2:17.2.7-915.g330067db.el8.x86_64 7/25 2024-04-24T22:25:00.532 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-ceph-argparse-2:17.2.7-915.g330067db.el8.x 8/25 2024-04-24T22:25:00.532 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-ceph-common-2:17.2.7-915.g330067db.el8.x86 9/25 2024-04-24T22:25:00.532 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 10/25 2024-04-24T22:25:00.532 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-rados-2:17.2.7-915.g330067db.el8.x86_64 11/25 2024-04-24T22:25:00.532 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 12/25 2024-04-24T22:25:00.532 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 13/25 2024-04-24T22:25:00.532 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-04-24T22:25:00.533 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-04-24T22:25:00.533 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-04-24T22:25:00.533 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-04-24T22:25:00.533 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-04-24T22:25:00.533 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-04-24T22:25:00.533 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-04-24T22:25:00.533 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-04-24T22:25:00.533 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librados2-2:17.2.7-915.g330067db.el8.x86_64 22/25 2024-04-24T22:25:00.533 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-24T22:25:00.533 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librbd1-2:17.2.7-915.g330067db.el8.x86_64 24/25 2024-04-24T22:25:00.715 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:55 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:25:01.425 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-24T22:25:01.425 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:01.425 INFO:teuthology.orchestra.run.smithi080.stdout:Upgraded: 2024-04-24T22:25:01.426 INFO:teuthology.orchestra.run.smithi080.stdout: librados2-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:01.426 INFO:teuthology.orchestra.run.smithi080.stdout: librbd1-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:01.426 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-04-24T22:25:01.426 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-base-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:01.426 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-common-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:01.426 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:01.426 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:01.426 INFO:teuthology.orchestra.run.smithi080.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-24T22:25:01.426 INFO:teuthology.orchestra.run.smithi080.stdout: libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:01.426 INFO:teuthology.orchestra.run.smithi080.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-24T22:25:01.426 INFO:teuthology.orchestra.run.smithi080.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-24T22:25:01.426 INFO:teuthology.orchestra.run.smithi080.stdout: libradosstriper1-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:01.426 INFO:teuthology.orchestra.run.smithi080.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-24T22:25:01.426 INFO:teuthology.orchestra.run.smithi080.stdout: librgw2-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:01.427 INFO:teuthology.orchestra.run.smithi080.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-24T22:25:01.427 INFO:teuthology.orchestra.run.smithi080.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-24T22:25:01.427 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ceph-argparse-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:01.427 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ceph-common-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:01.427 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:01.427 INFO:teuthology.orchestra.run.smithi080.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-24T22:25:01.427 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rados-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:01.427 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:01.427 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:01.427 INFO:teuthology.orchestra.run.smithi080.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-24T22:25:01.427 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:01.427 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:25:01.781 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-test 2024-04-24T22:25:02.330 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:25:02.335 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:25:02.335 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-04-24T22:25:02.335 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:25:02.335 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-24T22:25:02.335 INFO:teuthology.orchestra.run.smithi134.stdout: ceph x86_64 2:17.2.7-915.g330067db.el8 ceph 6.4 k 2024-04-24T22:25:02.336 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-04-24T22:25:02.336 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds x86_64 2:17.2.7-915.g330067db.el8 ceph 2.3 M 2024-04-24T22:25:02.336 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr x86_64 2:17.2.7-915.g330067db.el8 ceph 1.6 M 2024-04-24T22:25:02.336 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 240 k 2024-04-24T22:25:02.336 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon x86_64 2:17.2.7-915.g330067db.el8 ceph 4.0 M 2024-04-24T22:25:02.336 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd x86_64 2:17.2.7-915.g330067db.el8 ceph 17 M 2024-04-24T22:25:02.336 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-24T22:25:02.336 INFO:teuthology.orchestra.run.smithi134.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-24T22:25:02.337 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-24T22:25:02.337 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-24T22:25:02.337 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-24T22:25:02.337 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-24T22:25:02.337 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-24T22:25:02.337 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-24T22:25:02.337 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-24T22:25:02.338 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-24T22:25:02.338 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-24T22:25:02.338 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-24T22:25:02.338 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-24T22:25:02.338 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-24T22:25:02.338 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-24T22:25:02.339 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-24T22:25:02.339 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-24T22:25:02.339 INFO:teuthology.orchestra.run.smithi134.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-24T22:25:02.339 INFO:teuthology.orchestra.run.smithi134.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-24T22:25:02.339 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-24T22:25:02.339 INFO:teuthology.orchestra.run.smithi134.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-24T22:25:02.339 INFO:teuthology.orchestra.run.smithi134.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-24T22:25:02.339 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-24T22:25:02.339 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-24T22:25:02.340 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-24T22:25:02.340 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-24T22:25:02.340 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:25:02.340 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-24T22:25:02.340 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:25:02.340 INFO:teuthology.orchestra.run.smithi134.stdout:Install 32 Packages 2024-04-24T22:25:02.340 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:25:02.341 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 29 M 2024-04-24T22:25:02.341 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 113 M 2024-04-24T22:25:02.342 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-24T22:25:02.522 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:00:43 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:25:04.178 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:25:04.179 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:25:04.179 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-04-24T22:25:04.179 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:25:04.179 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-04-24T22:25:04.179 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-test x86_64 2:17.2.7-915.g330067db.el8 ceph 53 M 2024-04-24T22:25:04.179 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-04-24T22:25:04.179 INFO:teuthology.orchestra.run.smithi080.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-24T22:25:04.180 INFO:teuthology.orchestra.run.smithi080.stdout: libcephsqlite x86_64 2:17.2.7-915.g330067db.el8 ceph 179 k 2024-04-24T22:25:04.180 INFO:teuthology.orchestra.run.smithi080.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-24T22:25:04.180 INFO:teuthology.orchestra.run.smithi080.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-24T22:25:04.180 INFO:teuthology.orchestra.run.smithi080.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-24T22:25:04.180 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:04.180 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-04-24T22:25:04.180 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:25:04.180 INFO:teuthology.orchestra.run.smithi080.stdout:Install 6 Packages 2024-04-24T22:25:04.180 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:04.181 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 54 M 2024-04-24T22:25:04.181 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 236 M 2024-04-24T22:25:04.181 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-04-24T22:25:04.361 INFO:teuthology.orchestra.run.smithi134.stdout:(1/32): ceph-17.2.7-915.g330067db.el8.x86_64.rp 34 kB/s | 6.4 kB 00:00 2024-04-24T22:25:04.520 INFO:teuthology.orchestra.run.smithi134.stdout:(2/32): ceph-mds-17.2.7-915.g330067db.el8.x86_6 6.6 MB/s | 2.3 MB 00:00 2024-04-24T22:25:04.753 INFO:teuthology.orchestra.run.smithi134.stdout:(3/32): ceph-mgr-17.2.7-915.g330067db.el8.x86_6 2.7 MB/s | 1.6 MB 00:00 2024-04-24T22:25:04.879 INFO:teuthology.orchestra.run.smithi134.stdout:(4/32): ceph-mgr-modules-core-17.2.7-915.g33006 1.9 MB/s | 240 kB 00:00 2024-04-24T22:25:05.030 INFO:teuthology.orchestra.run.smithi080.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 1.0 MB/s | 203 kB 00:00 2024-04-24T22:25:05.071 INFO:teuthology.orchestra.run.smithi134.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 820 kB/s | 157 kB 00:00 2024-04-24T22:25:05.089 INFO:teuthology.orchestra.run.smithi080.stdout:(2/6): oniguruma-6.8.2-3.el8.x86_64.rpm 3.2 MB/s | 188 kB 00:00 2024-04-24T22:25:05.131 INFO:teuthology.orchestra.run.smithi134.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 669 kB/s | 39 kB 00:00 2024-04-24T22:25:05.147 INFO:teuthology.orchestra.run.smithi080.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2024-04-24T22:25:05.188 INFO:teuthology.orchestra.run.smithi134.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.7 MB/s | 103 kB 00:00 2024-04-24T22:25:05.206 INFO:teuthology.orchestra.run.smithi080.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-04-24T22:25:05.246 INFO:teuthology.orchestra.run.smithi134.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 979 kB/s | 57 kB 00:00 2024-04-24T22:25:05.306 INFO:teuthology.orchestra.run.smithi080.stdout:(5/6): libcephsqlite-17.2.7-915.g330067db.el8.x 380 kB/s | 179 kB 00:00 2024-04-24T22:25:05.438 INFO:teuthology.orchestra.run.smithi134.stdout:(9/32): ceph-osd-17.2.7-915.g330067db.el8.x86_6 19 MB/s | 17 MB 00:00 2024-04-24T22:25:05.472 INFO:teuthology.orchestra.run.smithi134.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.0 MB/s | 457 kB 00:00 2024-04-24T22:25:05.572 INFO:teuthology.orchestra.run.smithi134.stdout:(11/32): ceph-mon-17.2.7-915.g330067db.el8.x86_ 3.3 MB/s | 4.0 MB 00:01 2024-04-24T22:25:05.773 INFO:teuthology.orchestra.run.smithi134.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 745 kB/s | 249 kB 00:00 2024-04-24T22:25:05.831 INFO:teuthology.orchestra.run.smithi134.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 422 kB/s | 109 kB 00:00 2024-04-24T22:25:05.924 INFO:teuthology.orchestra.run.smithi134.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 290 kB/s | 44 kB 00:00 2024-04-24T22:25:05.949 INFO:teuthology.orchestra.run.smithi134.stdout:(15/32): python3-cryptography-3.2.1-7.el8.x86_6 1.3 MB/s | 641 kB 00:00 2024-04-24T22:25:05.999 INFO:teuthology.orchestra.run.smithi134.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.2 MB/s | 173 kB 00:00 2024-04-24T22:25:06.033 INFO:teuthology.orchestra.run.smithi134.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 924 kB/s | 185 kB 00:00 2024-04-24T22:25:06.050 INFO:teuthology.orchestra.run.smithi134.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 245 kB/s | 11 kB 00:00 2024-04-24T22:25:06.075 INFO:teuthology.orchestra.run.smithi134.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 425 kB/s | 18 kB 00:00 2024-04-24T22:25:06.092 INFO:teuthology.orchestra.run.smithi134.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 1.1 MB/s | 49 kB 00:00 2024-04-24T22:25:06.117 INFO:teuthology.orchestra.run.smithi134.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.4 MB/s | 59 kB 00:00 2024-04-24T22:25:06.159 INFO:teuthology.orchestra.run.smithi134.stdout:(22/32): python3-pecan-1.3.2-9.el8.noarch.rpm 4.1 MB/s | 283 kB 00:00 2024-04-24T22:25:06.176 INFO:teuthology.orchestra.run.smithi134.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 266 kB/s | 16 kB 00:00 2024-04-24T22:25:06.201 INFO:teuthology.orchestra.run.smithi134.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 447 kB/s | 19 kB 00:00 2024-04-24T22:25:06.227 INFO:teuthology.orchestra.run.smithi134.stdout:(25/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.4 MB/s | 384 kB 00:00 2024-04-24T22:25:06.244 INFO:teuthology.orchestra.run.smithi134.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 356 kB/s | 24 kB 00:00 2024-04-24T22:25:06.261 INFO:teuthology.orchestra.run.smithi134.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 498 kB/s | 29 kB 00:00 2024-04-24T22:25:06.278 INFO:teuthology.orchestra.run.smithi134.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 545 kB/s | 27 kB 00:00 2024-04-24T22:25:06.311 INFO:teuthology.orchestra.run.smithi134.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.9 MB/s | 251 kB 00:00 2024-04-24T22:25:06.336 INFO:teuthology.orchestra.run.smithi134.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.7 MB/s | 255 kB 00:00 2024-04-24T22:25:06.362 INFO:teuthology.orchestra.run.smithi134.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.0 MB/s | 85 kB 00:00 2024-04-24T22:25:06.387 INFO:teuthology.orchestra.run.smithi134.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 305 kB/s | 23 kB 00:00 2024-04-24T22:25:06.391 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:25:06.391 INFO:teuthology.orchestra.run.smithi134.stdout:Total 7.1 MB/s | 29 MB 00:04 2024-04-24T22:25:06.471 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-24T22:25:06.507 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-24T22:25:06.507 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-24T22:25:06.798 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-24T22:25:06.801 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-24T22:25:07.639 INFO:teuthology.orchestra.run.smithi080.stdout:(6/6): ceph-test-17.2.7-915.g330067db.el8.x86_6 19 MB/s | 53 MB 00:02 2024-04-24T22:25:07.641 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:25:07.642 INFO:teuthology.orchestra.run.smithi080.stdout:Total 15 MB/s | 54 MB 00:03 2024-04-24T22:25:07.683 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-04-24T22:25:07.703 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-04-24T22:25:07.703 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-04-24T22:25:07.895 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-24T22:25:08.018 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-04-24T22:25:08.021 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-04-24T22:25:08.174 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-24T22:25:08.375 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-24T22:25:08.483 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-04-24T22:25:08.601 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-04-24T22:25:08.691 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-04-24T22:25:08.857 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-04-24T22:25:08.870 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-04-24T22:25:09.005 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-24T22:25:09.072 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-04-24T22:25:09.110 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-24T22:25:09.269 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-04-24T22:25:09.304 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-04-24T22:25:09.445 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-04-24T22:25:09.455 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 5/6 2024-04-24T22:25:09.631 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-04-24T22:25:09.856 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-04-24T22:25:10.076 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-04-24T22:25:10.341 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-04-24T22:25:10.564 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-04-24T22:25:10.946 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-04-24T22:25:11.220 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-04-24T22:25:11.408 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-04-24T22:25:11.648 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-04-24T22:25:11.876 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-24T22:25:12.200 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-04-24T22:25:12.405 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-04-24T22:25:12.619 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-04-24T22:25:12.805 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-24T22:25:13.049 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-24T22:25:13.290 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-24T22:25:13.543 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-04-24T22:25:13.882 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-04-24T22:25:14.222 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-24T22:25:14.506 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 5/6 2024-04-24T22:25:14.594 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-915.g330067db.el8.n 27/32 2024-04-24T22:25:14.744 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-test-2:17.2.7-915.g330067db.el8.x86_64 6/6 2024-04-24T22:25:14.774 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 28/32 2024-04-24T22:25:15.024 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-test-2:17.2.7-915.g330067db.el8.x86_64 6/6 2024-04-24T22:25:15.024 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-test-2:17.2.7-915.g330067db.el8.x86_64 1/6 2024-04-24T22:25:15.024 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 2/6 2024-04-24T22:25:15.024 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-04-24T22:25:15.024 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-04-24T22:25:15.024 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-04-24T22:25:15.434 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-04-24T22:25:15.434 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:15.434 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-04-24T22:25:15.435 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-test-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:15.435 INFO:teuthology.orchestra.run.smithi080.stdout: jq-1.6-8.el8.x86_64 2024-04-24T22:25:15.435 INFO:teuthology.orchestra.run.smithi080.stdout: libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:15.435 INFO:teuthology.orchestra.run.smithi080.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-04-24T22:25:15.435 INFO:teuthology.orchestra.run.smithi080.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-24T22:25:15.435 INFO:teuthology.orchestra.run.smithi080.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-24T22:25:15.435 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:15.435 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:25:15.697 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph 2024-04-24T22:25:16.314 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 28/32 2024-04-24T22:25:16.434 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:00:57 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:25:16.507 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 29/32 2024-04-24T22:25:16.907 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 29/32 2024-04-24T22:25:17.108 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 30/32 2024-04-24T22:25:17.325 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 30/32 2024-04-24T22:25:18.075 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 31/32 2024-04-24T22:25:18.096 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 31/32 2024-04-24T22:25:18.099 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:25:18.103 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:25:18.103 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-04-24T22:25:18.103 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:25:18.103 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-04-24T22:25:18.103 INFO:teuthology.orchestra.run.smithi080.stdout: ceph x86_64 2:17.2.7-915.g330067db.el8 ceph 6.4 k 2024-04-24T22:25:18.103 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-04-24T22:25:18.103 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mds x86_64 2:17.2.7-915.g330067db.el8 ceph 2.3 M 2024-04-24T22:25:18.103 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr x86_64 2:17.2.7-915.g330067db.el8 ceph 1.6 M 2024-04-24T22:25:18.103 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-modules-core noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 240 k 2024-04-24T22:25:18.103 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mon x86_64 2:17.2.7-915.g330067db.el8 ceph 4.0 M 2024-04-24T22:25:18.103 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-osd x86_64 2:17.2.7-915.g330067db.el8 ceph 17 M 2024-04-24T22:25:18.103 INFO:teuthology.orchestra.run.smithi080.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-24T22:25:18.103 INFO:teuthology.orchestra.run.smithi080.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-24T22:25:18.104 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-24T22:25:18.104 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-24T22:25:18.104 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-24T22:25:18.104 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-24T22:25:18.104 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-24T22:25:18.104 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-24T22:25:18.104 INFO:teuthology.orchestra.run.smithi080.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-24T22:25:18.104 INFO:teuthology.orchestra.run.smithi080.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-24T22:25:18.104 INFO:teuthology.orchestra.run.smithi080.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-24T22:25:18.104 INFO:teuthology.orchestra.run.smithi080.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-24T22:25:18.104 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-24T22:25:18.104 INFO:teuthology.orchestra.run.smithi080.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-24T22:25:18.104 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-24T22:25:18.104 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-24T22:25:18.105 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-24T22:25:18.105 INFO:teuthology.orchestra.run.smithi080.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-24T22:25:18.105 INFO:teuthology.orchestra.run.smithi080.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-24T22:25:18.105 INFO:teuthology.orchestra.run.smithi080.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-24T22:25:18.105 INFO:teuthology.orchestra.run.smithi080.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-24T22:25:18.105 INFO:teuthology.orchestra.run.smithi080.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-24T22:25:18.105 INFO:teuthology.orchestra.run.smithi080.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-24T22:25:18.105 INFO:teuthology.orchestra.run.smithi080.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-24T22:25:18.105 INFO:teuthology.orchestra.run.smithi080.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-24T22:25:18.105 INFO:teuthology.orchestra.run.smithi080.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-24T22:25:18.105 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:18.105 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-04-24T22:25:18.105 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:25:18.105 INFO:teuthology.orchestra.run.smithi080.stdout:Install 32 Packages 2024-04-24T22:25:18.105 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:18.106 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 29 M 2024-04-24T22:25:18.106 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 113 M 2024-04-24T22:25:18.106 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-04-24T22:25:18.208 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-2:17.2.7-915.g330067db.el8.x86_64 32/32 2024-04-24T22:25:18.634 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-2:17.2.7-915.g330067db.el8.x86_64 32/32 2024-04-24T22:25:18.634 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-2:17.2.7-915.g330067db.el8.x86_64 1/32 2024-04-24T22:25:18.634 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 2/32 2024-04-24T22:25:18.634 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 3/32 2024-04-24T22:25:18.634 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 4/32 2024-04-24T22:25:18.634 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 5/32 2024-04-24T22:25:18.634 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-915.g330067db.el8.n 6/32 2024-04-24T22:25:18.634 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-04-24T22:25:18.634 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-04-24T22:25:18.634 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-04-24T22:25:18.635 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-04-24T22:25:18.635 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-04-24T22:25:18.635 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-04-24T22:25:18.635 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-04-24T22:25:18.635 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-04-24T22:25:18.635 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-04-24T22:25:18.635 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-04-24T22:25:18.635 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-24T22:25:18.635 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-04-24T22:25:18.635 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-04-24T22:25:18.635 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-04-24T22:25:18.635 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-04-24T22:25:18.635 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-04-24T22:25:18.635 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-04-24T22:25:18.635 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-04-24T22:25:18.636 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-04-24T22:25:18.636 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-04-24T22:25:18.636 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-04-24T22:25:18.636 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-04-24T22:25:18.636 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-04-24T22:25:18.636 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-04-24T22:25:18.636 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-04-24T22:25:18.799 INFO:teuthology.orchestra.run.smithi080.stdout:(1/32): ceph-17.2.7-915.g330067db.el8.x86_64.rp 36 kB/s | 6.4 kB 00:00 2024-04-24T22:25:18.941 INFO:teuthology.orchestra.run.smithi080.stdout:(2/32): ceph-mgr-17.2.7-915.g330067db.el8.x86_6 5.0 MB/s | 1.6 MB 00:00 2024-04-24T22:25:18.991 INFO:teuthology.orchestra.run.smithi080.stdout:(3/32): ceph-mds-17.2.7-915.g330067db.el8.x86_6 6.3 MB/s | 2.3 MB 00:00 2024-04-24T22:25:19.083 INFO:teuthology.orchestra.run.smithi080.stdout:(4/32): ceph-mgr-modules-core-17.2.7-915.g33006 2.6 MB/s | 240 kB 00:00 2024-04-24T22:25:19.292 INFO:teuthology.orchestra.run.smithi080.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 754 kB/s | 157 kB 00:00 2024-04-24T22:25:19.376 INFO:teuthology.orchestra.run.smithi080.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 467 kB/s | 39 kB 00:00 2024-04-24T22:25:19.434 INFO:teuthology.orchestra.run.smithi080.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.7 MB/s | 103 kB 00:00 2024-04-24T22:25:19.501 INFO:teuthology.orchestra.run.smithi080.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 857 kB/s | 57 kB 00:00 2024-04-24T22:25:19.533 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-04-24T22:25:19.534 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:25:19.534 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-24T22:25:19.534 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:19.534 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:19.534 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:19.534 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T22:25:19.534 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:19.534 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:19.534 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-24T22:25:19.534 INFO:teuthology.orchestra.run.smithi134.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-24T22:25:19.534 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-24T22:25:19.534 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-24T22:25:19.534 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-24T22:25:19.535 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-24T22:25:19.535 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-24T22:25:19.535 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-24T22:25:19.535 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-24T22:25:19.535 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-24T22:25:19.535 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-24T22:25:19.535 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-24T22:25:19.535 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-24T22:25:19.535 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend-2.6-1.el8.noarch 2024-04-24T22:25:19.535 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-24T22:25:19.535 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-24T22:25:19.535 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-24T22:25:19.535 INFO:teuthology.orchestra.run.smithi134.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-24T22:25:19.535 INFO:teuthology.orchestra.run.smithi134.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-24T22:25:19.535 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-24T22:25:19.536 INFO:teuthology.orchestra.run.smithi134.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-24T22:25:19.536 INFO:teuthology.orchestra.run.smithi134.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-24T22:25:19.536 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-24T22:25:19.536 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-24T22:25:19.536 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-24T22:25:19.536 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-24T22:25:19.536 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:25:19.536 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:25:19.627 INFO:teuthology.orchestra.run.smithi080.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 3.6 MB/s | 457 kB 00:00 2024-04-24T22:25:19.802 INFO:teuthology.orchestra.run.smithi080.stdout:(10/32): ceph-osd-17.2.7-915.g330067db.el8.x86_ 20 MB/s | 17 MB 00:00 2024-04-24T22:25:19.820 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-base 2024-04-24T22:25:20.127 INFO:teuthology.orchestra.run.smithi080.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 497 kB/s | 249 kB 00:00 2024-04-24T22:25:20.169 INFO:teuthology.orchestra.run.smithi080.stdout:(12/32): python3-cryptography-3.2.1-7.el8.x86_6 1.7 MB/s | 641 kB 00:00 2024-04-24T22:25:20.211 INFO:teuthology.orchestra.run.smithi080.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.3 MB/s | 109 kB 00:00 2024-04-24T22:25:20.420 INFO:teuthology.orchestra.run.smithi080.stdout:(14/32): ceph-mon-17.2.7-915.g330067db.el8.x86_ 2.5 MB/s | 4.0 MB 00:01 2024-04-24T22:25:20.445 INFO:teuthology.orchestra.run.smithi080.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 158 kB/s | 44 kB 00:00 2024-04-24T22:25:20.487 INFO:teuthology.orchestra.run.smithi080.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 673 kB/s | 185 kB 00:00 2024-04-24T22:25:20.554 INFO:teuthology.orchestra.run.smithi080.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.3 MB/s | 173 kB 00:00 2024-04-24T22:25:20.589 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:15 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:25:20.672 INFO:teuthology.orchestra.run.smithi080.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 60 kB/s | 11 kB 00:00 2024-04-24T22:25:20.728 INFO:teuthology.orchestra.run.smithi080.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 101 kB/s | 18 kB 00:00 2024-04-24T22:25:20.763 INFO:teuthology.orchestra.run.smithi080.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 530 kB/s | 49 kB 00:00 2024-04-24T22:25:20.805 INFO:teuthology.orchestra.run.smithi080.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 765 kB/s | 59 kB 00:00 2024-04-24T22:25:20.847 INFO:teuthology.orchestra.run.smithi080.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 957 kB/s | 384 kB 00:00 2024-04-24T22:25:20.914 INFO:teuthology.orchestra.run.smithi080.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 144 kB/s | 16 kB 00:00 2024-04-24T22:25:20.940 INFO:teuthology.orchestra.run.smithi080.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 204 kB/s | 19 kB 00:00 2024-04-24T22:25:20.964 INFO:teuthology.orchestra.run.smithi080.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2024-04-24T22:25:20.998 INFO:teuthology.orchestra.run.smithi080.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 285 kB/s | 24 kB 00:00 2024-04-24T22:25:21.023 INFO:teuthology.orchestra.run.smithi080.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 348 kB/s | 29 kB 00:00 2024-04-24T22:25:21.048 INFO:teuthology.orchestra.run.smithi080.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 328 kB/s | 27 kB 00:00 2024-04-24T22:25:21.124 INFO:teuthology.orchestra.run.smithi080.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.5 MB/s | 251 kB 00:00 2024-04-24T22:25:21.149 INFO:teuthology.orchestra.run.smithi080.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.7 MB/s | 255 kB 00:00 2024-04-24T22:25:21.174 INFO:teuthology.orchestra.run.smithi080.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 680 kB/s | 85 kB 00:00 2024-04-24T22:25:21.208 INFO:teuthology.orchestra.run.smithi080.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 275 kB/s | 23 kB 00:00 2024-04-24T22:25:21.209 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:25:21.209 INFO:teuthology.orchestra.run.smithi080.stdout:Total 9.3 MB/s | 29 MB 00:03 2024-04-24T22:25:21.288 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-04-24T22:25:21.325 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-04-24T22:25:21.326 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-04-24T22:25:21.627 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-04-24T22:25:21.630 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-04-24T22:25:22.188 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-base-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T22:25:22.235 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:25:22.236 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-04-24T22:25:22.236 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:25:22.347 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install cephadm 2024-04-24T22:25:22.722 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-04-24T22:25:22.958 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-24T22:25:23.102 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:18 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:25:23.168 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-24T22:25:23.418 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-04-24T22:25:23.624 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-04-24T22:25:23.848 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-04-24T22:25:24.028 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-04-24T22:25:24.221 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-04-24T22:25:24.366 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-04-24T22:25:24.574 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-04-24T22:25:24.776 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:25:24.776 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:25:24.776 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-04-24T22:25:24.776 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:25:24.776 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-24T22:25:24.776 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 82 k 2024-04-24T22:25:24.776 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:25:24.777 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-24T22:25:24.777 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:25:24.777 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-04-24T22:25:24.777 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:25:24.777 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 82 k 2024-04-24T22:25:24.777 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 372 k 2024-04-24T22:25:24.777 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-24T22:25:24.828 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-04-24T22:25:24.961 INFO:teuthology.orchestra.run.smithi134.stdout:cephadm-17.2.7-915.g330067db.el8.noarch.rpm 446 kB/s | 82 kB 00:00 2024-04-24T22:25:24.962 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:25:24.962 INFO:teuthology.orchestra.run.smithi134.stdout:Total 443 kB/s | 82 kB 00:00 2024-04-24T22:25:24.962 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-24T22:25:24.970 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-24T22:25:24.970 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-24T22:25:24.972 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-24T22:25:24.975 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-24T22:25:25.025 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-24T22:25:25.091 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-04-24T22:25:25.307 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-04-24T22:25:25.471 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T22:25:25.636 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephadm-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T22:25:25.673 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-04-24T22:25:25.861 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T22:25:25.929 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-04-24T22:25:26.101 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-04-24T22:25:26.219 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephadm-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T22:25:26.220 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:25:26.220 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-24T22:25:26.220 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T22:25:26.220 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:25:26.220 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:25:26.308 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-04-24T22:25:26.424 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-immutable-object-cache 2024-04-24T22:25:26.510 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-24T22:25:26.867 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-04-24T22:25:27.057 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-04-24T22:25:27.186 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:22 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:25:27.228 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-04-24T22:25:27.430 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-24T22:25:27.656 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-24T22:25:27.875 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-24T22:25:28.094 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-04-24T22:25:28.458 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-04-24T22:25:28.781 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-24T22:25:28.843 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:25:28.844 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:25:28.844 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-04-24T22:25:28.844 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:25:28.844 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-24T22:25:28.844 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-915.g330067db.el8 ceph 161 k 2024-04-24T22:25:28.845 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:25:28.845 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-24T22:25:28.845 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:25:28.845 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-04-24T22:25:28.845 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:25:28.845 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 161 k 2024-04-24T22:25:28.845 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 494 k 2024-04-24T22:25:28.845 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-24T22:25:29.146 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-915.g330067db.el8.n 27/32 2024-04-24T22:25:29.359 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 28/32 2024-04-24T22:25:29.921 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-immutable-object-cache-17.2.7-915.g330067d 150 kB/s | 161 kB 00:01 2024-04-24T22:25:29.921 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:25:29.922 INFO:teuthology.orchestra.run.smithi134.stdout:Total 150 kB/s | 161 kB 00:01 2024-04-24T22:25:29.922 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-24T22:25:29.934 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-24T22:25:29.935 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-24T22:25:29.986 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-24T22:25:29.990 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-24T22:25:30.201 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-24T22:25:30.373 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-915.g330067db.e 1/1 2024-04-24T22:25:30.794 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-915.g330067db.e 1/1 2024-04-24T22:25:30.892 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 28/32 2024-04-24T22:25:31.049 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 29/32 2024-04-24T22:25:31.174 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-915.g330067db.e 1/1 2024-04-24T22:25:31.174 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:25:31.174 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-24T22:25:31.174 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:31.174 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:25:31.174 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:25:31.395 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr 2024-04-24T22:25:31.447 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 29/32 2024-04-24T22:25:31.650 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 30/32 2024-04-24T22:25:31.867 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 30/32 2024-04-24T22:25:32.025 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 31/32 2024-04-24T22:25:32.042 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 31/32 2024-04-24T22:25:32.166 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-2:17.2.7-915.g330067db.el8.x86_64 32/32 2024-04-24T22:25:32.167 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:27 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:25:32.574 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-2:17.2.7-915.g330067db.el8.x86_64 32/32 2024-04-24T22:25:32.575 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-2:17.2.7-915.g330067db.el8.x86_64 1/32 2024-04-24T22:25:32.575 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 2/32 2024-04-24T22:25:32.575 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 3/32 2024-04-24T22:25:32.575 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 4/32 2024-04-24T22:25:32.575 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 5/32 2024-04-24T22:25:32.575 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-915.g330067db.el8.n 6/32 2024-04-24T22:25:32.575 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-04-24T22:25:32.575 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-04-24T22:25:32.575 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-04-24T22:25:32.575 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-04-24T22:25:32.575 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-04-24T22:25:32.575 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-04-24T22:25:32.575 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-04-24T22:25:32.575 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-04-24T22:25:32.576 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-04-24T22:25:32.576 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-04-24T22:25:32.576 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-24T22:25:32.576 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-04-24T22:25:32.576 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-04-24T22:25:32.576 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-04-24T22:25:32.576 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-04-24T22:25:32.576 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-04-24T22:25:32.576 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-04-24T22:25:32.576 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-04-24T22:25:32.576 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-04-24T22:25:32.576 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-04-24T22:25:32.576 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-04-24T22:25:32.576 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-04-24T22:25:32.576 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-04-24T22:25:32.577 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-04-24T22:25:32.577 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-04-24T22:25:33.766 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T22:25:33.814 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:25:33.815 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-04-24T22:25:33.815 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:25:33.938 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-dashboard 2024-04-24T22:25:34.220 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-04-24T22:25:34.221 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:34.221 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-04-24T22:25:34.221 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:34.221 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:34.221 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:34.221 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-modules-core-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T22:25:34.221 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:34.221 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:34.221 INFO:teuthology.orchestra.run.smithi080.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-24T22:25:34.221 INFO:teuthology.orchestra.run.smithi080.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-24T22:25:34.221 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-24T22:25:34.221 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-24T22:25:34.221 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-24T22:25:34.221 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-24T22:25:34.221 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-24T22:25:34.222 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-24T22:25:34.222 INFO:teuthology.orchestra.run.smithi080.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-24T22:25:34.222 INFO:teuthology.orchestra.run.smithi080.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-24T22:25:34.222 INFO:teuthology.orchestra.run.smithi080.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-24T22:25:34.222 INFO:teuthology.orchestra.run.smithi080.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-24T22:25:34.222 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-24T22:25:34.222 INFO:teuthology.orchestra.run.smithi080.stdout: python3-portend-2.6-1.el8.noarch 2024-04-24T22:25:34.222 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-24T22:25:34.222 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-24T22:25:34.222 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-24T22:25:34.222 INFO:teuthology.orchestra.run.smithi080.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-24T22:25:34.222 INFO:teuthology.orchestra.run.smithi080.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-24T22:25:34.222 INFO:teuthology.orchestra.run.smithi080.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-24T22:25:34.222 INFO:teuthology.orchestra.run.smithi080.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-24T22:25:34.223 INFO:teuthology.orchestra.run.smithi080.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-24T22:25:34.223 INFO:teuthology.orchestra.run.smithi080.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-24T22:25:34.223 INFO:teuthology.orchestra.run.smithi080.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-24T22:25:34.223 INFO:teuthology.orchestra.run.smithi080.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-24T22:25:34.223 INFO:teuthology.orchestra.run.smithi080.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-24T22:25:34.223 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:34.223 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:25:34.529 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-base 2024-04-24T22:25:34.696 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:29 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:25:35.283 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:16 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:25:36.350 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:25:36.351 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:25:36.351 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-04-24T22:25:36.351 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:25:36.351 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-24T22:25:36.352 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 3.4 M 2024-04-24T22:25:36.352 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-04-24T22:25:36.352 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 24 k 2024-04-24T22:25:36.352 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 14 k 2024-04-24T22:25:36.352 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-24T22:25:36.352 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-24T22:25:36.352 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-24T22:25:36.353 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:25:36.353 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-24T22:25:36.353 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:25:36.353 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2024-04-24T22:25:36.353 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:25:36.354 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 3.7 M 2024-04-24T22:25:36.354 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 80 M 2024-04-24T22:25:36.354 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-24T22:25:36.823 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): ceph-grafana-dashboards-17.2.7-915.g3300 168 kB/s | 24 kB 00:00 2024-04-24T22:25:36.901 INFO:teuthology.orchestra.run.smithi080.stdout:Package ceph-base-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T22:25:36.923 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): ceph-prometheus-alerts-17.2.7-915.g33006 59 kB/s | 14 kB 00:00 2024-04-24T22:25:36.950 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:25:36.951 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-04-24T22:25:36.951 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:25:36.998 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 448 kB/s | 34 kB 00:00 2024-04-24T22:25:37.057 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): ceph-mgr-dashboard-17.2.7-915.g330067db. 8.9 MB/s | 3.4 MB 00:00 2024-04-24T22:25:37.070 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install cephadm 2024-04-24T22:25:37.083 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 164 kB/s | 43 kB 00:00 2024-04-24T22:25:37.115 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.6 MB/s | 196 kB 00:00 2024-04-24T22:25:37.117 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:25:37.117 INFO:teuthology.orchestra.run.smithi134.stdout:Total 4.8 MB/s | 3.7 MB 00:00 2024-04-24T22:25:37.128 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-24T22:25:37.137 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-24T22:25:37.137 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-24T22:25:37.232 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-24T22:25:37.235 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-24T22:25:37.493 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-24T22:25:37.702 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-04-24T22:25:37.827 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:18 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:25:37.939 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-04-24T22:25:38.140 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-04-24T22:25:38.269 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-915.g330067db.el8.no 4/6 2024-04-24T22:25:39.376 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-915.g330067db.el8.n 5/6 2024-04-24T22:25:39.525 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:25:39.525 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:25:39.525 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-04-24T22:25:39.525 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:25:39.525 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-04-24T22:25:39.525 INFO:teuthology.orchestra.run.smithi080.stdout: cephadm noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 82 k 2024-04-24T22:25:39.526 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:39.526 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-04-24T22:25:39.526 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:25:39.526 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2024-04-24T22:25:39.526 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:39.526 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 82 k 2024-04-24T22:25:39.526 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 372 k 2024-04-24T22:25:39.527 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-04-24T22:25:39.603 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.noarch 6/6 2024-04-24T22:25:39.789 INFO:teuthology.orchestra.run.smithi080.stdout:cephadm-17.2.7-915.g330067db.el8.noarch.rpm 313 kB/s | 82 kB 00:00 2024-04-24T22:25:39.789 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:25:39.789 INFO:teuthology.orchestra.run.smithi080.stdout:Total 312 kB/s | 82 kB 00:00 2024-04-24T22:25:39.789 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-04-24T22:25:39.797 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-04-24T22:25:39.797 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-04-24T22:25:39.799 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-04-24T22:25:39.802 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-04-24T22:25:39.847 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-04-24T22:25:39.890 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.noarch 6/6 2024-04-24T22:25:39.890 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-915.g330067db.el8.n 1/6 2024-04-24T22:25:39.890 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.noarch 2/6 2024-04-24T22:25:39.890 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-915.g330067db.el8.no 3/6 2024-04-24T22:25:39.890 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-04-24T22:25:39.890 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-04-24T22:25:40.314 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-04-24T22:25:40.315 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:25:40.315 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-24T22:25:40.315 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T22:25:40.315 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T22:25:40.315 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T22:25:40.315 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-24T22:25:40.315 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-24T22:25:40.315 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-24T22:25:40.315 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:25:40.315 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:25:40.338 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: cephadm-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T22:25:40.446 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-24T22:25:40.485 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : cephadm-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T22:25:40.709 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: cephadm-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T22:25:41.107 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : cephadm-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T22:25:41.107 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:41.107 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-04-24T22:25:41.108 INFO:teuthology.orchestra.run.smithi080.stdout: cephadm-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T22:25:41.108 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:41.108 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:25:41.201 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:36 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:25:41.359 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-immutable-object-cache 2024-04-24T22:25:42.114 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:23 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:25:42.854 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:25:42.855 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-04-24T22:25:42.855 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-04-24T22:25:42.856 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-04-24T22:25:42.856 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-24T22:25:42.856 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 7.4 M 2024-04-24T22:25:42.856 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-04-24T22:25:42.856 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-04-24T22:25:42.856 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-04-24T22:25:42.856 INFO:teuthology.orchestra.run.smithi134.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-24T22:25:42.856 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-24T22:25:42.856 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-24T22:25:42.856 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-24T22:25:42.856 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-24T22:25:42.856 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:25:42.856 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-24T22:25:42.857 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-04-24T22:25:42.857 INFO:teuthology.orchestra.run.smithi134.stdout:Install 8 Packages 2024-04-24T22:25:42.857 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:25:42.857 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 36 M 2024-04-24T22:25:42.857 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 210 M 2024-04-24T22:25:42.857 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-24T22:25:43.597 INFO:teuthology.orchestra.run.smithi134.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2024-04-24T22:25:43.739 INFO:teuthology.orchestra.run.smithi134.stdout:(2/8): ceph-mgr-diskprediction-local-17.2.7-915 14 MB/s | 7.4 MB 00:00 2024-04-24T22:25:43.760 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:25:43.761 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:25:43.761 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repo Size 2024-04-24T22:25:43.761 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:25:43.761 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-04-24T22:25:43.761 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-915.g330067db.el8 ceph 161 k 2024-04-24T22:25:43.761 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:43.761 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-04-24T22:25:43.761 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:25:43.761 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2024-04-24T22:25:43.761 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:43.761 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 161 k 2024-04-24T22:25:43.761 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 494 k 2024-04-24T22:25:43.762 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-04-24T22:25:43.814 INFO:teuthology.orchestra.run.smithi134.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 17 MB/s | 3.7 MB 00:00 2024-04-24T22:25:43.889 INFO:teuthology.orchestra.run.smithi134.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 7.0 MB/s | 4.9 MB 00:00 2024-04-24T22:25:43.998 INFO:teuthology.orchestra.run.smithi134.stdout:(5/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 870 kB/s | 225 kB 00:00 2024-04-24T22:25:44.207 INFO:teuthology.orchestra.run.smithi134.stdout:(6/8): python3-scipy-1.0.0-21.module_el8+762+77 36 MB/s | 14 MB 00:00 2024-04-24T22:25:44.307 INFO:teuthology.orchestra.run.smithi134.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 559 kB/s | 172 kB 00:00 2024-04-24T22:25:44.365 INFO:teuthology.orchestra.run.smithi134.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.3 MB/s | 645 kB 00:00 2024-04-24T22:25:44.366 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:25:44.367 INFO:teuthology.orchestra.run.smithi134.stdout:Total 24 MB/s | 36 MB 00:01 2024-04-24T22:25:44.411 INFO:teuthology.orchestra.run.smithi080.stdout:ceph-immutable-object-cache-17.2.7-915.g330067d 249 kB/s | 161 kB 00:00 2024-04-24T22:25:44.412 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:25:44.412 INFO:teuthology.orchestra.run.smithi080.stdout:Total 248 kB/s | 161 kB 00:00 2024-04-24T22:25:44.413 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-04-24T22:25:44.424 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-04-24T22:25:44.424 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-04-24T22:25:44.474 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-04-24T22:25:44.477 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-04-24T22:25:44.597 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-24T22:25:44.615 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-24T22:25:44.615 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-24T22:25:45.013 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-24T22:25:45.016 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-24T22:25:45.107 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-04-24T22:25:45.606 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-24T22:25:46.205 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-915.g330067db.e 1/1 2024-04-24T22:25:46.566 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-24T22:25:46.605 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-915.g330067db.e 1/1 2024-04-24T22:25:47.002 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-915.g330067db.e 1/1 2024-04-24T22:25:47.002 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:47.002 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-04-24T22:25:47.002 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-immutable-object-cache-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:25:47.002 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:47.002 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:25:47.176 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-24T22:25:47.215 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-mgr 2024-04-24T22:25:47.328 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-24T22:25:47.925 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-24T22:25:47.979 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:28 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:25:48.119 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-04-24T22:25:48.722 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-04-24T22:25:48.861 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-24T22:25:49.556 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-24T22:25:49.590 INFO:teuthology.orchestra.run.smithi080.stdout:Package ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T22:25:49.640 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:25:49.641 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-04-24T22:25:49.641 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:25:49.765 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-mgr-dashboard 2024-04-24T22:25:49.914 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-24T22:25:50.169 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-24T22:25:50.515 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:31 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:25:51.929 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-24T22:25:52.150 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:25:52.151 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:25:52.151 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-04-24T22:25:52.151 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:25:52.151 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-04-24T22:25:52.151 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-dashboard noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 3.4 M 2024-04-24T22:25:52.151 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-04-24T22:25:52.151 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-grafana-dashboards noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 24 k 2024-04-24T22:25:52.151 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-prometheus-alerts noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 14 k 2024-04-24T22:25:52.152 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-24T22:25:52.152 INFO:teuthology.orchestra.run.smithi080.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-24T22:25:52.152 INFO:teuthology.orchestra.run.smithi080.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-24T22:25:52.152 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:52.152 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-04-24T22:25:52.152 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:25:52.152 INFO:teuthology.orchestra.run.smithi080.stdout:Install 6 Packages 2024-04-24T22:25:52.152 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:52.152 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 3.7 M 2024-04-24T22:25:52.153 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 80 M 2024-04-24T22:25:52.153 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-04-24T22:25:52.681 INFO:teuthology.orchestra.run.smithi080.stdout:(1/6): ceph-grafana-dashboards-17.2.7-915.g3300 163 kB/s | 24 kB 00:00 2024-04-24T22:25:52.805 INFO:teuthology.orchestra.run.smithi080.stdout:(2/6): ceph-prometheus-alerts-17.2.7-915.g33006 53 kB/s | 14 kB 00:00 2024-04-24T22:25:52.897 INFO:teuthology.orchestra.run.smithi080.stdout:(3/6): ceph-mgr-dashboard-17.2.7-915.g330067db. 9.2 MB/s | 3.4 MB 00:00 2024-04-24T22:25:52.922 INFO:teuthology.orchestra.run.smithi080.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 176 kB/s | 43 kB 00:00 2024-04-24T22:25:52.947 INFO:teuthology.orchestra.run.smithi080.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 236 kB/s | 34 kB 00:00 2024-04-24T22:25:53.122 INFO:teuthology.orchestra.run.smithi080.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 873 kB/s | 196 kB 00:00 2024-04-24T22:25:53.124 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:25:53.124 INFO:teuthology.orchestra.run.smithi080.stdout:Total 3.8 MB/s | 3.7 MB 00:00 2024-04-24T22:25:53.134 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-04-24T22:25:53.134 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-04-24T22:25:53.143 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-04-24T22:25:53.143 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-04-24T22:25:53.231 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-04-24T22:25:53.234 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-04-24T22:25:53.311 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-915.g330067db 8/8 2024-04-24T22:25:53.523 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-04-24T22:25:53.614 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-915.g330067db 8/8 2024-04-24T22:25:53.614 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-915.g330067db 1/8 2024-04-24T22:25:53.614 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-04-24T22:25:53.614 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-04-24T22:25:53.614 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-24T22:25:53.614 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-24T22:25:53.614 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-04-24T22:25:53.615 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-04-24T22:25:53.695 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-04-24T22:25:53.914 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-04-24T22:25:54.040 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-04-24T22:25:54.041 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:25:54.041 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-24T22:25:54.041 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T22:25:54.041 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-04-24T22:25:54.041 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-04-24T22:25:54.041 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-24T22:25:54.041 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-24T22:25:54.041 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-24T22:25:54.041 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-24T22:25:54.041 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-24T22:25:54.041 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:25:54.041 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:25:54.081 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-04-24T22:25:54.172 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-rook 2024-04-24T22:25:54.218 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-915.g330067db.el8.no 4/6 2024-04-24T22:25:54.931 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:49 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:25:55.309 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-915.g330067db.el8.n 5/6 2024-04-24T22:25:55.560 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.noarch 6/6 2024-04-24T22:25:55.836 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.noarch 6/6 2024-04-24T22:25:55.837 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-915.g330067db.el8.n 1/6 2024-04-24T22:25:55.837 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.noarch 2/6 2024-04-24T22:25:55.837 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-915.g330067db.el8.no 3/6 2024-04-24T22:25:55.838 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-04-24T22:25:55.838 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-04-24T22:25:56.213 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-04-24T22:25:56.213 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:56.213 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-04-24T22:25:56.214 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-grafana-dashboards-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T22:25:56.214 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T22:25:56.214 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-prometheus-alerts-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T22:25:56.214 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-24T22:25:56.214 INFO:teuthology.orchestra.run.smithi080.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-24T22:25:56.214 INFO:teuthology.orchestra.run.smithi080.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-24T22:25:56.214 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:56.214 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:25:56.348 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-24T22:25:56.570 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:25:56.572 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:25:56.572 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-04-24T22:25:56.572 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:25:56.572 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-24T22:25:56.572 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 47 k 2024-04-24T22:25:56.572 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-04-24T22:25:56.572 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-24T22:25:56.572 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-24T22:25:56.572 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-24T22:25:56.572 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-04-24T22:25:56.572 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-04-24T22:25:56.573 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-24T22:25:56.573 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-24T22:25:56.573 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-24T22:25:56.573 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-24T22:25:56.573 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-24T22:25:56.573 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-24T22:25:56.573 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-24T22:25:56.573 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:25:56.573 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-24T22:25:56.573 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:25:56.573 INFO:teuthology.orchestra.run.smithi134.stdout:Install 13 Packages 2024-04-24T22:25:56.573 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:25:56.574 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 1.9 M 2024-04-24T22:25:56.574 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 25 M 2024-04-24T22:25:56.574 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-24T22:25:57.097 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:38 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:25:57.423 INFO:teuthology.orchestra.run.smithi134.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 204 kB/s | 27 kB 00:00 2024-04-24T22:25:57.448 INFO:teuthology.orchestra.run.smithi134.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 129 kB/s | 20 kB 00:00 2024-04-24T22:25:57.557 INFO:teuthology.orchestra.run.smithi134.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 948 kB/s | 126 kB 00:00 2024-04-24T22:25:57.582 INFO:teuthology.orchestra.run.smithi134.stdout:(4/13): ceph-mgr-rook-17.2.7-915.g330067db.el8. 164 kB/s | 47 kB 00:00 2024-04-24T22:25:57.607 INFO:teuthology.orchestra.run.smithi134.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 692 kB/s | 110 kB 00:00 2024-04-24T22:25:57.741 INFO:teuthology.orchestra.run.smithi134.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 247 kB/s | 33 kB 00:00 2024-04-24T22:25:57.792 INFO:teuthology.orchestra.run.smithi134.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 204 kB/s | 43 kB 00:00 2024-04-24T22:25:57.817 INFO:teuthology.orchestra.run.smithi134.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 217 kB/s | 16 kB 00:00 2024-04-24T22:25:57.876 INFO:teuthology.orchestra.run.smithi134.stdout:(9/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 487 kB/s | 155 kB 00:00 2024-04-24T22:25:57.952 INFO:teuthology.orchestra.run.smithi134.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 511 kB/s | 81 kB 00:00 2024-04-24T22:25:58.010 INFO:teuthology.orchestra.run.smithi134.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 481 kB/s | 64 kB 00:00 2024-04-24T22:25:58.037 INFO:teuthology.orchestra.run.smithi134.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 735 kB/s | 61 kB 00:00 2024-04-24T22:25:58.069 INFO:teuthology.orchestra.run.smithi134.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.6 MB/s | 1.2 MB 00:00 2024-04-24T22:25:58.069 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:25:58.072 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2024-04-24T22:25:58.126 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-24T22:25:58.146 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-24T22:25:58.146 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-24T22:25:58.231 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-24T22:25:58.233 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-24T22:25:58.729 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:25:58.729 INFO:teuthology.orchestra.run.smithi080.stdout:============================================================================================ 2024-04-24T22:25:58.730 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-04-24T22:25:58.730 INFO:teuthology.orchestra.run.smithi080.stdout:============================================================================================ 2024-04-24T22:25:58.730 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-04-24T22:25:58.730 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 7.4 M 2024-04-24T22:25:58.730 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-04-24T22:25:58.730 INFO:teuthology.orchestra.run.smithi080.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-04-24T22:25:58.730 INFO:teuthology.orchestra.run.smithi080.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-04-24T22:25:58.730 INFO:teuthology.orchestra.run.smithi080.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-24T22:25:58.730 INFO:teuthology.orchestra.run.smithi080.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-24T22:25:58.730 INFO:teuthology.orchestra.run.smithi080.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-24T22:25:58.730 INFO:teuthology.orchestra.run.smithi080.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-24T22:25:58.730 INFO:teuthology.orchestra.run.smithi080.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-24T22:25:58.730 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:58.730 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-04-24T22:25:58.731 INFO:teuthology.orchestra.run.smithi080.stdout:============================================================================================ 2024-04-24T22:25:58.731 INFO:teuthology.orchestra.run.smithi080.stdout:Install 8 Packages 2024-04-24T22:25:58.731 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:25:58.731 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 36 M 2024-04-24T22:25:58.731 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 210 M 2024-04-24T22:25:58.732 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-04-24T22:25:58.811 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-24T22:25:59.049 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-04-24T22:25:59.225 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-04-24T22:25:59.272 INFO:teuthology.orchestra.run.smithi080.stdout:(1/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 12 MB/s | 4.9 MB 00:00 2024-04-24T22:25:59.339 INFO:teuthology.orchestra.run.smithi080.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 9.8 MB/s | 4.7 MB 00:00 2024-04-24T22:25:59.431 INFO:teuthology.orchestra.run.smithi080.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 24 MB/s | 3.7 MB 00:00 2024-04-24T22:25:59.456 INFO:teuthology.orchestra.run.smithi080.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.9 MB/s | 225 kB 00:00 2024-04-24T22:25:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-04-24T22:25:59.708 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-04-24T22:25:59.773 INFO:teuthology.orchestra.run.smithi080.stdout:(5/8): python3-scipy-1.0.0-21.module_el8+762+77 41 MB/s | 14 MB 00:00 2024-04-24T22:25:59.865 INFO:teuthology.orchestra.run.smithi080.stdout:(6/8): ceph-mgr-diskprediction-local-17.2.7-915 7.4 MB/s | 7.4 MB 00:01 2024-04-24T22:25:59.923 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-04-24T22:25:59.998 INFO:teuthology.orchestra.run.smithi080.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.2 MB/s | 645 kB 00:00 2024-04-24T22:26:00.117 INFO:teuthology.orchestra.run.smithi080.stdout:(8/8): libquadmath-8.5.0-21.el8.x86_64.rpm 504 kB/s | 172 kB 00:00 2024-04-24T22:26:00.117 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:26:00.117 INFO:teuthology.orchestra.run.smithi080.stdout:Total 26 MB/s | 36 MB 00:01 2024-04-24T22:26:00.195 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-04-24T22:26:00.345 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-04-24T22:26:00.364 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-04-24T22:26:00.364 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-04-24T22:26:00.487 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-04-24T22:26:00.737 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-04-24T22:26:00.745 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-04-24T22:26:00.748 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-04-24T22:26:01.325 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-04-24T22:26:01.493 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-24T22:26:01.622 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-24T22:26:01.777 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-24T22:26:01.786 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-04-24T22:26:02.065 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-04-24T22:26:02.308 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-04-24T22:26:02.539 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-24T22:26:02.576 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-04-24T22:26:02.750 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 13/13 2024-04-24T22:26:02.977 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-04-24T22:26:03.033 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 13/13 2024-04-24T22:26:03.033 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 1/13 2024-04-24T22:26:03.033 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-04-24T22:26:03.033 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-04-24T22:26:03.033 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-04-24T22:26:03.033 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-04-24T22:26:03.033 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-04-24T22:26:03.033 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-04-24T22:26:03.033 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-04-24T22:26:03.033 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-04-24T22:26:03.034 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-04-24T22:26:03.034 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-04-24T22:26:03.034 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-04-24T22:26:03.527 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-04-24T22:26:03.527 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:03.527 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-24T22:26:03.527 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T22:26:03.527 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-24T22:26:03.527 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-24T22:26:03.527 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-24T22:26:03.527 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-24T22:26:03.528 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-24T22:26:03.528 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-24T22:26:03.528 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-24T22:26:03.528 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-24T22:26:03.528 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-24T22:26:03.528 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-24T22:26:03.528 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-24T22:26:03.528 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-24T22:26:03.528 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:03.528 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:26:03.598 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-04-24T22:26:03.656 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-cephadm 2024-04-24T22:26:03.743 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-24T22:26:04.419 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:59 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:26:04.431 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-24T22:26:04.763 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-24T22:26:05.042 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-24T22:26:06.073 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:26:06.074 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:06.074 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-04-24T22:26:06.075 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:06.075 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-24T22:26:06.075 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 123 k 2024-04-24T22:26:06.075 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-04-24T22:26:06.075 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-24T22:26:06.075 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-04-24T22:26:06.075 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-04-24T22:26:06.076 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-24T22:26:06.076 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:06.076 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-24T22:26:06.076 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:06.076 INFO:teuthology.orchestra.run.smithi134.stdout:Install 5 Packages 2024-04-24T22:26:06.076 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:06.077 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 6.0 M 2024-04-24T22:26:06.077 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 26 M 2024-04-24T22:26:06.078 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-24T22:26:06.576 INFO:teuthology.orchestra.run.smithi134.stdout:(1/5): python3-jinja2-2.10.1-4.el8.noarch.rpm 2.0 MB/s | 578 kB 00:00 2024-04-24T22:26:06.710 INFO:teuthology.orchestra.run.smithi134.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 12 MB/s | 4.8 MB 00:00 2024-04-24T22:26:06.744 INFO:teuthology.orchestra.run.smithi134.stdout:(3/5): ceph-mgr-cephadm-17.2.7-915.g330067db.el 276 kB/s | 123 kB 00:00 2024-04-24T22:26:06.807 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-24T22:26:07.335 INFO:teuthology.orchestra.run.smithi134.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 97 kB/s | 60 kB 00:00 2024-04-24T22:26:07.744 INFO:teuthology.orchestra.run.smithi134.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 408 kB/s | 476 kB 00:01 2024-04-24T22:26:07.746 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:26:07.747 INFO:teuthology.orchestra.run.smithi134.stdout:Total 3.6 MB/s | 6.0 MB 00:01 2024-04-24T22:26:07.801 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-24T22:26:07.813 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-24T22:26:07.813 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-24T22:26:07.916 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-24T22:26:07.919 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-24T22:26:08.045 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-04-24T22:26:08.183 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-24T22:26:08.225 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-915.g330067db 8/8 2024-04-24T22:26:08.531 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-915.g330067db 8/8 2024-04-24T22:26:08.531 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-915.g330067db 1/8 2024-04-24T22:26:08.531 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-04-24T22:26:08.531 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-04-24T22:26:08.531 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-24T22:26:08.531 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-24T22:26:08.531 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-04-24T22:26:08.531 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-04-24T22:26:08.539 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2024-04-24T22:26:08.936 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-04-24T22:26:08.936 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:08.936 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-04-24T22:26:08.936 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-diskprediction-local-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T22:26:08.936 INFO:teuthology.orchestra.run.smithi080.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-04-24T22:26:08.936 INFO:teuthology.orchestra.run.smithi080.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-04-24T22:26:08.936 INFO:teuthology.orchestra.run.smithi080.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-24T22:26:08.937 INFO:teuthology.orchestra.run.smithi080.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-24T22:26:08.937 INFO:teuthology.orchestra.run.smithi080.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-24T22:26:08.937 INFO:teuthology.orchestra.run.smithi080.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-24T22:26:08.937 INFO:teuthology.orchestra.run.smithi080.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-24T22:26:08.937 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:08.937 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:26:09.081 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-mgr-rook 2024-04-24T22:26:09.511 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-24T22:26:09.835 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:01:50 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:26:09.861 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2024-04-24T22:26:10.177 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 4/5 2024-04-24T22:26:10.382 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noarch 5/5 2024-04-24T22:26:10.669 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noarch 5/5 2024-04-24T22:26:10.669 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noarch 1/5 2024-04-24T22:26:10.669 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2024-04-24T22:26:10.669 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/5 2024-04-24T22:26:10.669 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-24T22:26:11.061 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2024-04-24T22:26:11.061 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:11.062 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-24T22:26:11.062 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T22:26:11.062 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-24T22:26:11.062 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-24T22:26:11.062 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-04-24T22:26:11.062 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-24T22:26:11.062 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:11.062 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:26:11.185 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-fuse 2024-04-24T22:26:11.483 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:26:11.484 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:26:11.484 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-04-24T22:26:11.484 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:26:11.484 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-04-24T22:26:11.485 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-rook noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 47 k 2024-04-24T22:26:11.485 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-04-24T22:26:11.485 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-24T22:26:11.485 INFO:teuthology.orchestra.run.smithi080.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-24T22:26:11.485 INFO:teuthology.orchestra.run.smithi080.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-24T22:26:11.485 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-04-24T22:26:11.485 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-04-24T22:26:11.485 INFO:teuthology.orchestra.run.smithi080.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-24T22:26:11.485 INFO:teuthology.orchestra.run.smithi080.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-24T22:26:11.485 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-24T22:26:11.485 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-24T22:26:11.485 INFO:teuthology.orchestra.run.smithi080.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-24T22:26:11.485 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-24T22:26:11.485 INFO:teuthology.orchestra.run.smithi080.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-24T22:26:11.486 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:11.486 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-04-24T22:26:11.486 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:26:11.486 INFO:teuthology.orchestra.run.smithi080.stdout:Install 13 Packages 2024-04-24T22:26:11.486 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:11.486 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 1.9 M 2024-04-24T22:26:11.486 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 25 M 2024-04-24T22:26:11.487 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-04-24T22:26:11.941 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:06 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:26:12.180 INFO:teuthology.orchestra.run.smithi080.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 169 kB/s | 20 kB 00:00 2024-04-24T22:26:12.197 INFO:teuthology.orchestra.run.smithi080.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 197 kB/s | 27 kB 00:00 2024-04-24T22:26:12.214 INFO:teuthology.orchestra.run.smithi080.stdout:(3/13): ceph-mgr-rook-17.2.7-915.g330067db.el8. 312 kB/s | 47 kB 00:00 2024-04-24T22:26:12.315 INFO:teuthology.orchestra.run.smithi080.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 940 kB/s | 110 kB 00:00 2024-04-24T22:26:12.348 INFO:teuthology.orchestra.run.smithi080.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 754 kB/s | 126 kB 00:00 2024-04-24T22:26:12.524 INFO:teuthology.orchestra.run.smithi080.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 502 kB/s | 155 kB 00:00 2024-04-24T22:26:12.540 INFO:teuthology.orchestra.run.smithi080.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 189 kB/s | 43 kB 00:00 2024-04-24T22:26:12.574 INFO:teuthology.orchestra.run.smithi080.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 146 kB/s | 33 kB 00:00 2024-04-24T22:26:12.775 INFO:teuthology.orchestra.run.smithi080.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 65 kB/s | 16 kB 00:00 2024-04-24T22:26:12.833 INFO:teuthology.orchestra.run.smithi080.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 278 kB/s | 81 kB 00:00 2024-04-24T22:26:12.917 INFO:teuthology.orchestra.run.smithi080.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 734 kB/s | 61 kB 00:00 2024-04-24T22:26:12.942 INFO:teuthology.orchestra.run.smithi080.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 384 kB/s | 64 kB 00:00 2024-04-24T22:26:12.976 INFO:teuthology.orchestra.run.smithi080.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.9 MB/s | 1.2 MB 00:00 2024-04-24T22:26:12.976 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:26:12.976 INFO:teuthology.orchestra.run.smithi080.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2024-04-24T22:26:13.034 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-04-24T22:26:13.049 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-04-24T22:26:13.050 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-04-24T22:26:13.135 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-04-24T22:26:13.138 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-04-24T22:26:13.558 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:26:13.559 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:13.559 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-04-24T22:26:13.559 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:13.559 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-24T22:26:13.559 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse x86_64 2:17.2.7-915.g330067db.el8 ceph 876 k 2024-04-24T22:26:13.559 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-04-24T22:26:13.559 INFO:teuthology.orchestra.run.smithi134.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-24T22:26:13.560 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:13.560 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-24T22:26:13.560 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:13.560 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2024-04-24T22:26:13.560 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:13.560 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 959 k 2024-04-24T22:26:13.560 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 3.1 M 2024-04-24T22:26:13.560 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-24T22:26:13.720 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-04-24T22:26:13.886 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): fuse-2.9.7-19.el8.x86_64.rpm 313 kB/s | 83 kB 00:00 2024-04-24T22:26:13.963 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-04-24T22:26:14.027 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): ceph-fuse-17.2.7-915.g330067db.el8.x86_6 2.1 MB/s | 876 kB 00:00 2024-04-24T22:26:14.028 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:26:14.028 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.0 MB/s | 959 kB 00:00 2024-04-24T22:26:14.038 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-24T22:26:14.059 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-24T22:26:14.059 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-24T22:26:14.145 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-24T22:26:14.148 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-24T22:26:14.149 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-04-24T22:26:14.318 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-24T22:26:14.384 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-04-24T22:26:14.583 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-24T22:26:14.622 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-04-24T22:26:14.807 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 2/2 2024-04-24T22:26:14.812 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-04-24T22:26:15.061 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-04-24T22:26:15.196 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 2/2 2024-04-24T22:26:15.196 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 1/2 2024-04-24T22:26:15.327 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-04-24T22:26:15.553 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-04-24T22:26:15.608 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-24T22:26:15.608 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:15.608 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-24T22:26:15.608 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-24T22:26:15.609 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:15.609 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:26:15.843 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-volume 2024-04-24T22:26:16.565 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-04-24T22:26:16.614 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:11 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:26:16.855 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-04-24T22:26:17.047 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-04-24T22:26:17.240 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-04-24T22:26:17.415 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 13/13 2024-04-24T22:26:17.742 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 13/13 2024-04-24T22:26:17.742 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 1/13 2024-04-24T22:26:17.742 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-04-24T22:26:17.742 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-04-24T22:26:17.742 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-04-24T22:26:17.742 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-04-24T22:26:17.742 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-04-24T22:26:17.742 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-04-24T22:26:17.742 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-04-24T22:26:17.743 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-04-24T22:26:17.743 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-04-24T22:26:17.743 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-04-24T22:26:17.743 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-04-24T22:26:18.203 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-04-24T22:26:18.203 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:18.204 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-04-24T22:26:18.204 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T22:26:18.204 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-24T22:26:18.204 INFO:teuthology.orchestra.run.smithi080.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-24T22:26:18.204 INFO:teuthology.orchestra.run.smithi080.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-24T22:26:18.204 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-24T22:26:18.204 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-24T22:26:18.204 INFO:teuthology.orchestra.run.smithi080.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-24T22:26:18.204 INFO:teuthology.orchestra.run.smithi080.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-24T22:26:18.204 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-24T22:26:18.204 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-24T22:26:18.204 INFO:teuthology.orchestra.run.smithi080.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-24T22:26:18.204 INFO:teuthology.orchestra.run.smithi080.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-24T22:26:18.204 INFO:teuthology.orchestra.run.smithi080.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-24T22:26:18.204 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:18.205 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:26:18.275 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:26:18.276 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:18.276 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-04-24T22:26:18.276 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:18.276 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-24T22:26:18.276 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 267 k 2024-04-24T22:26:18.276 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:18.276 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-24T22:26:18.276 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:18.276 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-04-24T22:26:18.276 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:18.277 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 267 k 2024-04-24T22:26:18.277 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 1.3 M 2024-04-24T22:26:18.277 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-24T22:26:18.322 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-mgr-cephadm 2024-04-24T22:26:19.089 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:00 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:26:19.396 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-volume-17.2.7-915.g330067db.el8.noarch.rpm 239 kB/s | 267 kB 00:01 2024-04-24T22:26:19.397 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:26:19.397 INFO:teuthology.orchestra.run.smithi134.stdout:Total 239 kB/s | 267 kB 00:01 2024-04-24T22:26:19.398 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-24T22:26:19.409 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-24T22:26:19.409 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-24T22:26:19.465 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-24T22:26:19.468 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-24T22:26:19.764 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-24T22:26:20.007 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-volume-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T22:26:20.411 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-volume-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T22:26:20.750 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:26:20.751 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:26:20.751 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-04-24T22:26:20.751 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:26:20.751 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-04-24T22:26:20.751 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-cephadm noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 123 k 2024-04-24T22:26:20.751 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-04-24T22:26:20.751 INFO:teuthology.orchestra.run.smithi080.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-24T22:26:20.751 INFO:teuthology.orchestra.run.smithi080.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-04-24T22:26:20.751 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-04-24T22:26:20.751 INFO:teuthology.orchestra.run.smithi080.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-24T22:26:20.751 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:20.752 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-04-24T22:26:20.752 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:26:20.752 INFO:teuthology.orchestra.run.smithi080.stdout:Install 5 Packages 2024-04-24T22:26:20.752 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:20.752 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 6.0 M 2024-04-24T22:26:20.752 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 26 M 2024-04-24T22:26:20.752 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-04-24T22:26:20.810 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-volume-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T22:26:20.810 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:20.810 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-24T22:26:20.810 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T22:26:20.811 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:20.811 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:26:21.086 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados-devel 2024-04-24T22:26:21.491 INFO:teuthology.orchestra.run.smithi080.stdout:(1/5): ceph-mgr-cephadm-17.2.7-915.g330067db.el 658 kB/s | 123 kB 00:00 2024-04-24T22:26:21.583 INFO:teuthology.orchestra.run.smithi080.stdout:(2/5): python3-jinja2-2.10.1-4.el8.noarch.rpm 2.0 MB/s | 578 kB 00:00 2024-04-24T22:26:21.733 INFO:teuthology.orchestra.run.smithi080.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 11 MB/s | 4.8 MB 00:00 2024-04-24T22:26:21.791 INFO:teuthology.orchestra.run.smithi080.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 289 kB/s | 60 kB 00:00 2024-04-24T22:26:21.858 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:16 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:26:21.875 INFO:teuthology.orchestra.run.smithi080.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.2 MB/s | 476 kB 00:00 2024-04-24T22:26:21.876 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:26:21.876 INFO:teuthology.orchestra.run.smithi080.stdout:Total 5.3 MB/s | 6.0 MB 00:01 2024-04-24T22:26:21.931 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-04-24T22:26:21.942 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-04-24T22:26:21.942 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-04-24T22:26:22.045 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-04-24T22:26:22.048 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-04-24T22:26:22.317 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-04-24T22:26:22.670 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2024-04-24T22:26:23.491 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:26:23.491 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:23.492 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-04-24T22:26:23.492 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:23.492 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-24T22:26:23.492 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel x86_64 2:17.2.7-915.g330067db.el8 ceph 130 k 2024-04-24T22:26:23.492 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:23.492 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-24T22:26:23.492 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:23.492 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-04-24T22:26:23.492 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:23.493 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 130 k 2024-04-24T22:26:23.493 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 464 k 2024-04-24T22:26:23.493 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-24T22:26:23.625 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-24T22:26:23.902 INFO:teuthology.orchestra.run.smithi134.stdout:librados-devel-17.2.7-915.g330067db.el8.x86_64. 317 kB/s | 130 kB 00:00 2024-04-24T22:26:23.902 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:26:23.903 INFO:teuthology.orchestra.run.smithi134.stdout:Total 316 kB/s | 130 kB 00:00 2024-04-24T22:26:23.903 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-24T22:26:23.908 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-24T22:26:23.908 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-24T22:26:23.957 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-24T22:26:23.961 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2024-04-24T22:26:23.962 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-24T22:26:24.078 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-24T22:26:24.266 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 4/5 2024-04-24T22:26:24.304 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:26:24.447 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noarch 5/5 2024-04-24T22:26:24.591 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:26:24.740 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noarch 5/5 2024-04-24T22:26:24.740 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noarch 1/5 2024-04-24T22:26:24.740 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2024-04-24T22:26:24.740 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/5 2024-04-24T22:26:24.740 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-24T22:26:24.967 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:26:24.967 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:24.967 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-24T22:26:24.967 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:26:24.967 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:24.967 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:26:25.131 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2024-04-24T22:26:25.131 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:25.131 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-04-24T22:26:25.131 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T22:26:25.132 INFO:teuthology.orchestra.run.smithi080.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-24T22:26:25.132 INFO:teuthology.orchestra.run.smithi080.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-24T22:26:25.132 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-04-24T22:26:25.132 INFO:teuthology.orchestra.run.smithi080.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-24T22:26:25.132 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:25.132 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:26:25.269 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-fuse 2024-04-24T22:26:26.027 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:07 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:26:26.341 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs2 2024-04-24T22:26:27.108 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:22 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:26:27.684 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:26:27.685 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:26:27.685 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-04-24T22:26:27.685 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:26:27.685 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-04-24T22:26:27.685 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-fuse x86_64 2:17.2.7-915.g330067db.el8 ceph 876 k 2024-04-24T22:26:27.685 INFO:teuthology.orchestra.run.smithi080.stdout:Installing dependencies: 2024-04-24T22:26:27.686 INFO:teuthology.orchestra.run.smithi080.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-24T22:26:27.686 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:27.686 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-04-24T22:26:27.686 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:26:27.686 INFO:teuthology.orchestra.run.smithi080.stdout:Install 2 Packages 2024-04-24T22:26:27.686 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:27.687 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 959 k 2024-04-24T22:26:27.687 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 3.1 M 2024-04-24T22:26:27.687 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-04-24T22:26:28.216 INFO:teuthology.orchestra.run.smithi080.stdout:(1/2): fuse-2.9.7-19.el8.x86_64.rpm 202 kB/s | 83 kB 00:00 2024-04-24T22:26:28.705 INFO:teuthology.orchestra.run.smithi134.stdout:Package libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T22:26:28.753 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:26:28.754 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-04-24T22:26:28.755 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:26:28.875 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs-devel 2024-04-24T22:26:29.041 INFO:teuthology.orchestra.run.smithi080.stdout:(2/2): ceph-fuse-17.2.7-915.g330067db.el8.x86_6 708 kB/s | 876 kB 00:01 2024-04-24T22:26:29.042 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:26:29.043 INFO:teuthology.orchestra.run.smithi080.stdout:Total 707 kB/s | 959 kB 00:01 2024-04-24T22:26:29.055 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-04-24T22:26:29.076 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-04-24T22:26:29.076 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-04-24T22:26:29.165 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-04-24T22:26:29.168 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-04-24T22:26:29.332 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-04-24T22:26:29.604 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-24T22:26:29.627 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:24 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:26:29.795 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 2/2 2024-04-24T22:26:30.194 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 2/2 2024-04-24T22:26:30.194 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 1/2 2024-04-24T22:26:30.590 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-24T22:26:30.590 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:30.590 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-04-24T22:26:30.590 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-24T22:26:30.591 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:30.591 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:26:30.808 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install ceph-volume 2024-04-24T22:26:31.277 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:26:31.278 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:31.279 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-04-24T22:26:31.279 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:31.279 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-24T22:26:31.279 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel x86_64 2:17.2.7-915.g330067db.el8 ceph 25 k 2024-04-24T22:26:31.279 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:31.279 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-24T22:26:31.279 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:31.279 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-04-24T22:26:31.279 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:31.279 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 25 k 2024-04-24T22:26:31.279 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 109 k 2024-04-24T22:26:31.279 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-24T22:26:31.571 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:12 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:26:31.590 INFO:teuthology.orchestra.run.smithi134.stdout:libcephfs-devel-17.2.7-915.g330067db.el8.x86_64 80 kB/s | 25 kB 00:00 2024-04-24T22:26:31.590 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:26:31.590 INFO:teuthology.orchestra.run.smithi134.stdout:Total 80 kB/s | 25 kB 00:00 2024-04-24T22:26:31.591 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-24T22:26:31.594 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-24T22:26:31.594 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-24T22:26:31.596 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-24T22:26:31.599 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-24T22:26:31.782 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-24T22:26:32.320 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:26:32.601 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:26:32.947 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:26:32.948 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:32.948 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-24T22:26:32.948 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:26:32.948 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:32.948 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:26:33.074 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados2 2024-04-24T22:26:33.226 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:26:33.227 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:26:33.227 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repository Size 2024-04-24T22:26:33.227 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:26:33.227 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-04-24T22:26:33.227 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-volume noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 267 k 2024-04-24T22:26:33.227 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:33.228 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-04-24T22:26:33.228 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:26:33.228 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2024-04-24T22:26:33.228 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:33.228 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 267 k 2024-04-24T22:26:33.228 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 1.3 M 2024-04-24T22:26:33.228 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-04-24T22:26:33.439 INFO:teuthology.orchestra.run.smithi080.stdout:ceph-volume-17.2.7-915.g330067db.el8.noarch.rpm 1.2 MB/s | 267 kB 00:00 2024-04-24T22:26:33.440 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:26:33.440 INFO:teuthology.orchestra.run.smithi080.stdout:Total 1.2 MB/s | 267 kB 00:00 2024-04-24T22:26:33.440 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-04-24T22:26:33.450 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-04-24T22:26:33.450 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-04-24T22:26:33.486 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-04-24T22:26:33.489 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-04-24T22:26:33.664 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-04-24T22:26:33.829 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:28 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:26:33.887 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : ceph-volume-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T22:26:34.287 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: ceph-volume-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T22:26:35.123 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : ceph-volume-2:17.2.7-915.g330067db.el8.noarch 1/1 2024-04-24T22:26:35.124 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:35.124 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-04-24T22:26:35.124 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-volume-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T22:26:35.124 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:35.124 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:26:35.394 INFO:teuthology.orchestra.run.smithi134.stdout:Package librados2-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T22:26:35.410 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install librados-devel 2024-04-24T22:26:35.442 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:26:35.443 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-04-24T22:26:35.444 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:26:35.552 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librbd1 2024-04-24T22:26:36.175 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:17 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:26:36.346 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:31 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:26:37.831 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:26:37.831 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:26:37.832 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repo Size 2024-04-24T22:26:37.832 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:26:37.832 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-04-24T22:26:37.832 INFO:teuthology.orchestra.run.smithi080.stdout: librados-devel x86_64 2:17.2.7-915.g330067db.el8 ceph 130 k 2024-04-24T22:26:37.832 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:37.832 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-04-24T22:26:37.832 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:26:37.832 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2024-04-24T22:26:37.832 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:37.832 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 130 k 2024-04-24T22:26:37.832 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 464 k 2024-04-24T22:26:37.833 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-04-24T22:26:37.951 INFO:teuthology.orchestra.run.smithi134.stdout:Package librbd1-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T22:26:38.000 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:26:38.001 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-04-24T22:26:38.001 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:26:38.115 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rados 2024-04-24T22:26:38.260 INFO:teuthology.orchestra.run.smithi080.stdout:librados-devel-17.2.7-915.g330067db.el8.x86_64. 303 kB/s | 130 kB 00:00 2024-04-24T22:26:38.261 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:26:38.261 INFO:teuthology.orchestra.run.smithi080.stdout:Total 302 kB/s | 130 kB 00:00 2024-04-24T22:26:38.261 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-04-24T22:26:38.266 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-04-24T22:26:38.267 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-04-24T22:26:38.315 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-04-24T22:26:38.318 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-04-24T22:26:38.470 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-04-24T22:26:38.701 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : librados-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:26:38.865 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:33 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:26:38.975 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: librados-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:26:39.379 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : librados-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:26:39.379 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:39.379 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-04-24T22:26:39.379 INFO:teuthology.orchestra.run.smithi080.stdout: librados-devel-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:26:39.379 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:39.379 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:26:39.594 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install libcephfs2 2024-04-24T22:26:40.349 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:21 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:26:40.460 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rados-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T22:26:40.507 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:26:40.509 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-04-24T22:26:40.509 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:26:40.620 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rgw 2024-04-24T22:26:41.370 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:36 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:26:41.955 INFO:teuthology.orchestra.run.smithi080.stdout:Package libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T22:26:42.002 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:26:42.003 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-04-24T22:26:42.003 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:26:42.122 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install libcephfs-devel 2024-04-24T22:26:42.871 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:23 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:26:42.992 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T22:26:43.040 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:26:43.041 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-04-24T22:26:43.041 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:26:43.152 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-cephfs 2024-04-24T22:26:43.915 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:38 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:26:44.516 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:26:44.517 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:26:44.517 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repo Size 2024-04-24T22:26:44.517 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:26:44.517 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-04-24T22:26:44.517 INFO:teuthology.orchestra.run.smithi080.stdout: libcephfs-devel x86_64 2:17.2.7-915.g330067db.el8 ceph 25 k 2024-04-24T22:26:44.517 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:44.517 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-04-24T22:26:44.518 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:26:44.518 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2024-04-24T22:26:44.518 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:44.518 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 25 k 2024-04-24T22:26:44.518 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 109 k 2024-04-24T22:26:44.518 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-04-24T22:26:44.666 INFO:teuthology.orchestra.run.smithi080.stdout:libcephfs-devel-17.2.7-915.g330067db.el8.x86_64 168 kB/s | 25 kB 00:00 2024-04-24T22:26:44.667 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:26:44.667 INFO:teuthology.orchestra.run.smithi080.stdout:Total 166 kB/s | 25 kB 00:00 2024-04-24T22:26:44.667 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-04-24T22:26:44.670 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-04-24T22:26:44.671 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-04-24T22:26:44.672 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-04-24T22:26:44.675 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-04-24T22:26:44.725 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-04-24T22:26:44.916 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:26:45.166 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:26:45.513 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:26:45.514 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:45.514 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-04-24T22:26:45.514 INFO:teuthology.orchestra.run.smithi080.stdout: libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:26:45.514 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:26:45.514 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:26:45.523 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T22:26:45.571 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:26:45.572 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-04-24T22:26:45.572 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:26:45.645 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install librados2 2024-04-24T22:26:45.689 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rbd 2024-04-24T22:26:46.391 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:27 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:26:46.440 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:41 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:26:47.990 INFO:teuthology.orchestra.run.smithi080.stdout:Package librados2-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T22:26:48.037 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:26:48.038 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-04-24T22:26:48.038 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:26:48.042 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T22:26:48.090 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:26:48.091 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-04-24T22:26:48.091 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:26:48.152 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install librbd1 2024-04-24T22:26:48.200 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-fuse 2024-04-24T22:26:48.907 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:29 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:26:48.950 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:43 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:26:50.516 INFO:teuthology.orchestra.run.smithi080.stdout:Package librbd1-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T22:26:50.563 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:26:50.564 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-04-24T22:26:50.564 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:26:50.580 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:26:50.580 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:50.580 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-04-24T22:26:50.580 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:50.581 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-24T22:26:50.581 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse x86_64 2:17.2.7-915.g330067db.el8 ceph 91 k 2024-04-24T22:26:50.581 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:50.581 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-24T22:26:50.581 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:50.581 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-04-24T22:26:50.581 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:50.582 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 91 k 2024-04-24T22:26:50.582 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 268 k 2024-04-24T22:26:50.582 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-24T22:26:50.680 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install python3-rados 2024-04-24T22:26:50.996 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-fuse-17.2.7-915.g330067db.el8.x86_64.rpm 219 kB/s | 91 kB 00:00 2024-04-24T22:26:50.996 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:26:50.997 INFO:teuthology.orchestra.run.smithi134.stdout:Total 218 kB/s | 91 kB 00:00 2024-04-24T22:26:50.997 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-24T22:26:51.006 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-24T22:26:51.006 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-24T22:26:51.055 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-24T22:26:51.058 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-24T22:26:51.157 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-24T22:26:51.358 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:26:51.428 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:32 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:26:51.653 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:26:52.012 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:26:52.012 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:52.012 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-24T22:26:52.012 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:26:52.012 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:52.012 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:26:52.228 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-mirror 2024-04-24T22:26:52.987 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:47 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:26:53.015 INFO:teuthology.orchestra.run.smithi080.stdout:Package python3-rados-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T22:26:53.062 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:26:53.064 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-04-24T22:26:53.064 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:26:53.181 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install python3-rgw 2024-04-24T22:26:53.926 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:34 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:26:54.637 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:26:54.638 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:54.638 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-04-24T22:26:54.638 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:54.638 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-24T22:26:54.638 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror x86_64 2:17.2.7-915.g330067db.el8 ceph 3.5 M 2024-04-24T22:26:54.639 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:54.639 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-24T22:26:54.639 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:54.639 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-04-24T22:26:54.639 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:54.639 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 3.5 M 2024-04-24T22:26:54.639 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 16 M 2024-04-24T22:26:54.639 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-24T22:26:55.206 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-mirror-17.2.7-915.g330067db.el8.x86_64.rpm 6.1 MB/s | 3.5 MB 00:00 2024-04-24T22:26:55.207 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:26:55.207 INFO:teuthology.orchestra.run.smithi134.stdout:Total 6.1 MB/s | 3.5 MB 00:00 2024-04-24T22:26:55.207 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-24T22:26:55.219 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-24T22:26:55.219 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-24T22:26:55.280 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-24T22:26:55.283 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-24T22:26:55.501 INFO:teuthology.orchestra.run.smithi080.stdout:Package python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T22:26:55.549 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:26:55.550 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-04-24T22:26:55.550 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:26:55.672 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install python3-cephfs 2024-04-24T22:26:55.715 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-24T22:26:55.925 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:26:56.351 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:26:56.426 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:37 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:26:56.723 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:26:56.723 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:56.723 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-24T22:26:56.723 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:26:56.723 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:56.723 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:26:56.923 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-nbd 2024-04-24T22:26:57.691 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:52 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:26:58.029 INFO:teuthology.orchestra.run.smithi080.stdout:Package python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T22:26:58.076 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:26:58.078 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-04-24T22:26:58.078 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:26:58.202 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install python3-rbd 2024-04-24T22:26:58.946 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:39 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:26:59.332 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:26:59.333 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:59.333 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-04-24T22:26:59.333 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:59.333 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-04-24T22:26:59.333 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd x86_64 2:17.2.7-915.g330067db.el8 ceph 175 k 2024-04-24T22:26:59.333 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:59.333 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-04-24T22:26:59.333 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-04-24T22:26:59.333 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-04-24T22:26:59.333 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:26:59.334 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 175 k 2024-04-24T22:26:59.334 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 556 k 2024-04-24T22:26:59.334 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-04-24T22:26:59.642 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-nbd-17.2.7-915.g330067db.el8.x86_64.rpm 567 kB/s | 175 kB 00:00 2024-04-24T22:26:59.643 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:26:59.643 INFO:teuthology.orchestra.run.smithi134.stdout:Total 565 kB/s | 175 kB 00:00 2024-04-24T22:26:59.643 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-04-24T22:26:59.654 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-04-24T22:26:59.654 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-04-24T22:26:59.704 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-04-24T22:26:59.707 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-04-24T22:26:59.832 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-04-24T22:27:00.024 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:27:00.317 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:27:00.546 INFO:teuthology.orchestra.run.smithi080.stdout:Package python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T22:27:00.593 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:27:00.594 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-04-24T22:27:00.594 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:27:00.665 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:27:00.665 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:00.665 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-04-24T22:27:00.665 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:27:00.665 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:00.665 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:27:00.716 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install rbd-fuse 2024-04-24T22:27:00.885 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-mirror 2024-04-24T22:27:01.466 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:42 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:27:01.654 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:56 ago on Wed 24 Apr 2024 10:24:05 PM UTC. 2024-04-24T22:27:03.114 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:27:03.115 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:27:03.115 INFO:teuthology.orchestra.run.smithi080.stdout: Package Architecture Version Repository Size 2024-04-24T22:27:03.115 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:27:03.115 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-04-24T22:27:03.115 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-fuse x86_64 2:17.2.7-915.g330067db.el8 ceph 91 k 2024-04-24T22:27:03.115 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:03.115 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-04-24T22:27:03.115 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:27:03.115 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2024-04-24T22:27:03.115 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:03.116 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 91 k 2024-04-24T22:27:03.116 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 268 k 2024-04-24T22:27:03.116 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-04-24T22:27:03.229 INFO:teuthology.orchestra.run.smithi134.stdout:Package rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T22:27:03.277 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-04-24T22:27:03.279 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-04-24T22:27:03.279 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-04-24T22:27:03.393 DEBUG:teuthology.parallel:result is None 2024-04-24T22:27:03.397 INFO:teuthology.orchestra.run.smithi080.stdout:rbd-fuse-17.2.7-915.g330067db.el8.x86_64.rpm 323 kB/s | 91 kB 00:00 2024-04-24T22:27:03.398 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:27:03.398 INFO:teuthology.orchestra.run.smithi080.stdout:Total 321 kB/s | 91 kB 00:00 2024-04-24T22:27:03.398 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-04-24T22:27:03.407 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-04-24T22:27:03.407 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-04-24T22:27:03.456 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-04-24T22:27:03.459 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-04-24T22:27:03.566 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-04-24T22:27:03.754 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:27:04.030 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:27:04.409 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:27:04.410 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:04.410 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-04-24T22:27:04.410 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:27:04.410 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:04.410 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:27:04.620 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install rbd-mirror 2024-04-24T22:27:05.383 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:46 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:27:07.029 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:27:07.029 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:27:07.029 INFO:teuthology.orchestra.run.smithi080.stdout: Package Arch Version Repo Size 2024-04-24T22:27:07.029 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:27:07.030 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-04-24T22:27:07.030 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-mirror x86_64 2:17.2.7-915.g330067db.el8 ceph 3.5 M 2024-04-24T22:27:07.030 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:07.030 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-04-24T22:27:07.030 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:27:07.030 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2024-04-24T22:27:07.030 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:07.031 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 3.5 M 2024-04-24T22:27:07.031 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 16 M 2024-04-24T22:27:07.031 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-04-24T22:27:07.967 INFO:teuthology.orchestra.run.smithi080.stdout:rbd-mirror-17.2.7-915.g330067db.el8.x86_64.rpm 3.7 MB/s | 3.5 MB 00:00 2024-04-24T22:27:07.967 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:27:07.967 INFO:teuthology.orchestra.run.smithi080.stdout:Total 3.7 MB/s | 3.5 MB 00:00 2024-04-24T22:27:07.967 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-04-24T22:27:07.978 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-04-24T22:27:07.978 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-04-24T22:27:08.042 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-04-24T22:27:08.043 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-04-24T22:27:08.480 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-04-24T22:27:08.662 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:27:09.075 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:27:09.459 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:27:09.460 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:09.460 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-04-24T22:27:09.460 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:27:09.460 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:09.460 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:27:09.668 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install rbd-nbd 2024-04-24T22:27:10.435 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:51 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:27:12.076 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:27:12.076 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:27:12.076 INFO:teuthology.orchestra.run.smithi080.stdout: Package Architecture Version Repository Size 2024-04-24T22:27:12.076 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:27:12.076 INFO:teuthology.orchestra.run.smithi080.stdout:Installing: 2024-04-24T22:27:12.076 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-nbd x86_64 2:17.2.7-915.g330067db.el8 ceph 175 k 2024-04-24T22:27:12.076 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:12.076 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction Summary 2024-04-24T22:27:12.077 INFO:teuthology.orchestra.run.smithi080.stdout:================================================================================ 2024-04-24T22:27:12.077 INFO:teuthology.orchestra.run.smithi080.stdout:Install 1 Package 2024-04-24T22:27:12.077 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:12.077 INFO:teuthology.orchestra.run.smithi080.stdout:Total download size: 175 k 2024-04-24T22:27:12.077 INFO:teuthology.orchestra.run.smithi080.stdout:Installed size: 556 k 2024-04-24T22:27:12.077 INFO:teuthology.orchestra.run.smithi080.stdout:Downloading Packages: 2024-04-24T22:27:12.376 INFO:teuthology.orchestra.run.smithi080.stdout:rbd-nbd-17.2.7-915.g330067db.el8.x86_64.rpm 584 kB/s | 175 kB 00:00 2024-04-24T22:27:12.377 INFO:teuthology.orchestra.run.smithi080.stdout:-------------------------------------------------------------------------------- 2024-04-24T22:27:12.377 INFO:teuthology.orchestra.run.smithi080.stdout:Total 582 kB/s | 175 kB 00:00 2024-04-24T22:27:12.377 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction check 2024-04-24T22:27:12.388 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction check succeeded. 2024-04-24T22:27:12.388 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction test 2024-04-24T22:27:12.437 INFO:teuthology.orchestra.run.smithi080.stdout:Transaction test succeeded. 2024-04-24T22:27:12.440 INFO:teuthology.orchestra.run.smithi080.stdout:Running transaction 2024-04-24T22:27:12.591 INFO:teuthology.orchestra.run.smithi080.stdout: Preparing : 1/1 2024-04-24T22:27:12.768 INFO:teuthology.orchestra.run.smithi080.stdout: Installing : rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:27:13.038 INFO:teuthology.orchestra.run.smithi080.stdout: Running scriptlet: rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:27:13.454 INFO:teuthology.orchestra.run.smithi080.stdout: Verifying : rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 1/1 2024-04-24T22:27:13.454 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:13.454 INFO:teuthology.orchestra.run.smithi080.stdout:Installed: 2024-04-24T22:27:13.454 INFO:teuthology.orchestra.run.smithi080.stdout: rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T22:27:13.454 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:13.454 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:27:13.661 DEBUG:teuthology.orchestra.run.smithi080:> sudo yum -y install rbd-mirror 2024-04-24T22:27:14.423 INFO:teuthology.orchestra.run.smithi080.stdout:Last metadata expiration check: 0:02:55 ago on Wed 24 Apr 2024 10:24:19 PM UTC. 2024-04-24T22:27:16.041 INFO:teuthology.orchestra.run.smithi080.stdout:Package rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 is already installed. 2024-04-24T22:27:16.090 INFO:teuthology.orchestra.run.smithi080.stdout:Dependencies resolved. 2024-04-24T22:27:16.091 INFO:teuthology.orchestra.run.smithi080.stdout:Nothing to do. 2024-04-24T22:27:16.091 INFO:teuthology.orchestra.run.smithi080.stdout:Complete! 2024-04-24T22:27:16.201 DEBUG:teuthology.parallel:result is None 2024-04-24T22:27:16.202 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T22:27:16.426 DEBUG:teuthology.orchestra.run.smithi080:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T22:27:16.449 INFO:teuthology.orchestra.run.smithi080.stdout:17.2.7-915.g330067db.el8 2024-04-24T22:27:16.449 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-915.g330067db.el8 2024-04-24T22:27:16.450 INFO:teuthology.task.install:The correct ceph version 17.2.7-915.g330067db is installed. 2024-04-24T22:27:16.451 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T22:27:16.598 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T22:27:16.624 INFO:teuthology.orchestra.run.smithi134.stdout:17.2.7-915.g330067db.el8 2024-04-24T22:27:16.625 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-915.g330067db.el8 2024-04-24T22:27:16.625 INFO:teuthology.task.install:The correct ceph version 17.2.7-915.g330067db is installed. 2024-04-24T22:27:16.626 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-24T22:27:16.627 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:27:16.627 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T22:27:16.654 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:27:16.654 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T22:27:16.697 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-24T22:27:16.697 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:27:16.698 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T22:27:16.724 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T22:27:16.793 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:27:16.793 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T22:27:16.824 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T22:27:16.894 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-24T22:27:16.895 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:27:16.895 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T22:27:16.921 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T22:27:16.989 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:27:16.989 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T22:27:17.018 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T22:27:17.087 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-24T22:27:17.087 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:27:17.087 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T22:27:17.113 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T22:27:17.183 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:27:17.183 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T22:27:17.212 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T22:27:17.281 INFO:teuthology.run_tasks:Running task ceph... 2024-04-24T22:27:17.367 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-24T22:27:17.367 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 777 /var/log/ceph 2024-04-24T22:27:17.369 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 777 /var/log/ceph 2024-04-24T22:27:17.397 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-24T22:27:17.398 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-24T22:27:17.436 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-24T22:27:17.466 INFO:tasks.ceph:Creating extra log directories... 2024-04-24T22:27:17.467 DEBUG:teuthology.orchestra.run.smithi080:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-24T22:27:17.503 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-24T22:27:17.551 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-04-24T22:27:17.551 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-24T22:27:17.552 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-24_21:25:53-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7672255', 'branch': 'wip-yuriw4-testing-20240423.151325-quincy', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': None, 'first_in_suite': False, 'job_id': '7672255', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2024-04-24_21:25:53-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuriw4-testing-20240423.151325-quincy'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuriw4-testing-20240423.151325-quincy', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9954, 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-yuriw4-testing-20240423.151325-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_330067dbce47655d1f762ef14d546d2fde16a6a9/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9', 'targets': {'smithi080.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFtIISksWdrVRHufao/k3yWyq3S/2VFVLmirMvq9bK3yYzjhKiiias2g1SEIIOlD4G0INN8vwRO7Xj2+HQW6lo0=', 'smithi134.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBHEp9XcSD+LRpP+1zpkogwKIriYSdDjcXeYzLZCqzOirEGLV4Wd+CCGg/fGq3EI+YNtR9rEpDMqf2Rhx3KfpQQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-24_21:25:53', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-24T22:27:17.552 DEBUG:teuthology.orchestra.run.smithi080:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-04-24T22:27:17.570 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-04-24T22:27:17.597 DEBUG:teuthology.orchestra.run.smithi080:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T22:27:17.628 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T22:27:17.664 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:27:17.664 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T22:27:17.709 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T22:27:17.709 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_1 2024-04-24T22:27:17.767 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T22:27:17.767 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T22:27:17.767 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 6h/6d Inode: 163421 Links: 1 2024-04-24T22:27:17.768 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T22:27:17.768 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T22:27:17.768 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-04-24 22:27:08.832028001 +0000 2024-04-24T22:27:17.768 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-04-24 22:23:13.277370884 +0000 2024-04-24T22:27:17.768 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-04-24 22:23:13.277370884 +0000 2024-04-24T22:27:17.768 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: - 2024-04-24T22:27:17.768 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T22:27:17.833 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-04-24T22:27:17.833 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-04-24T22:27:17.833 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000270707 s, 1.9 MB/s 2024-04-24T22:27:17.834 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T22:27:17.893 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_2 2024-04-24T22:27:17.951 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T22:27:17.951 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T22:27:17.951 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 6h/6d Inode: 167891 Links: 1 2024-04-24T22:27:17.951 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T22:27:17.951 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T22:27:17.952 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-04-24 22:27:08.832028001 +0000 2024-04-24T22:27:17.952 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-04-24 22:23:13.605363298 +0000 2024-04-24T22:27:17.952 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-04-24 22:23:13.605363298 +0000 2024-04-24T22:27:17.952 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: - 2024-04-24T22:27:17.952 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T22:27:18.017 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-04-24T22:27:18.017 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-04-24T22:27:18.017 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000235707 s, 2.2 MB/s 2024-04-24T22:27:18.018 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T22:27:18.077 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_3 2024-04-24T22:27:18.135 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T22:27:18.135 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T22:27:18.135 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 6h/6d Inode: 168337 Links: 1 2024-04-24T22:27:18.136 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T22:27:18.136 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T22:27:18.136 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-04-24 22:27:08.832028001 +0000 2024-04-24T22:27:18.136 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-04-24 22:23:13.945355434 +0000 2024-04-24T22:27:18.136 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-04-24 22:23:13.945355434 +0000 2024-04-24T22:27:18.136 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: - 2024-04-24T22:27:18.136 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T22:27:18.203 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-04-24T22:27:18.203 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-04-24T22:27:18.203 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.000179311 s, 2.9 MB/s 2024-04-24T22:27:18.204 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T22:27:18.263 DEBUG:teuthology.orchestra.run.smithi080:> stat /dev/vg_nvme/lv_4 2024-04-24T22:27:18.321 INFO:teuthology.orchestra.run.smithi080.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T22:27:18.321 INFO:teuthology.orchestra.run.smithi080.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T22:27:18.321 INFO:teuthology.orchestra.run.smithi080.stdout:Device: 6h/6d Inode: 163575 Links: 1 2024-04-24T22:27:18.321 INFO:teuthology.orchestra.run.smithi080.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T22:27:18.321 INFO:teuthology.orchestra.run.smithi080.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T22:27:18.321 INFO:teuthology.orchestra.run.smithi080.stdout:Access: 2024-04-24 22:27:08.832028001 +0000 2024-04-24T22:27:18.321 INFO:teuthology.orchestra.run.smithi080.stdout:Modify: 2024-04-24 22:23:14.288347501 +0000 2024-04-24T22:27:18.322 INFO:teuthology.orchestra.run.smithi080.stdout:Change: 2024-04-24 22:23:14.288347501 +0000 2024-04-24T22:27:18.322 INFO:teuthology.orchestra.run.smithi080.stdout: Birth: - 2024-04-24T22:27:18.322 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T22:27:18.386 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records in 2024-04-24T22:27:18.386 INFO:teuthology.orchestra.run.smithi080.stderr:1+0 records out 2024-04-24T22:27:18.386 INFO:teuthology.orchestra.run.smithi080.stderr:512 bytes copied, 0.00020722 s, 2.5 MB/s 2024-04-24T22:27:18.387 DEBUG:teuthology.orchestra.run.smithi080:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T22:27:18.446 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T22:27:18.446 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi080.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-24T22:27:18.446 INFO:tasks.ceph:Generating config... 2024-04-24T22:27:18.447 INFO:tasks.ceph:[client] debug rbd = 20 2024-04-24T22:27:18.447 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-04-24T22:27:18.447 INFO:tasks.ceph:[client] log to stderr = False 2024-04-24T22:27:18.447 INFO:tasks.ceph:[client] rbd default features = 125 2024-04-24T22:27:18.448 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2024-04-24T22:27:18.448 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:18.448 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:18.448 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:18.448 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:18.448 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:18.448 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:18.448 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:18.448 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:18.448 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-24T22:27:18.448 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-04-24T22:27:18.448 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-04-24T22:27:18.448 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-24T22:27:18.449 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-24T22:27:18.449 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-24T22:27:18.449 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-24T22:27:18.449 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-24T22:27:18.449 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-24T22:27:18.449 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2024-04-24T22:27:18.449 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-24T22:27:18.449 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-24T22:27:18.449 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-24T22:27:18.449 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-24T22:27:18.449 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-24T22:27:18.449 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-24T22:27:18.449 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-24T22:27:18.449 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-24T22:27:18.449 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-24T22:27:18.450 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-24T22:27:18.450 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-24T22:27:18.450 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-24T22:27:18.450 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-04-24T22:27:18.450 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-04-24T22:27:18.532 INFO:teuthology.orchestra.run.smithi080.stdout:creating /etc/ceph/cluster1.keyring 2024-04-24T22:27:18.534 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-04-24T22:27:18.575 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-04-24T22:27:18.641 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.80')] 2024-04-24T22:27:18.642 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.80', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-04-24T22:27:18.642 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:27:18.642 DEBUG:teuthology.orchestra.run.smithi080:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T22:27:18.699 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.80 --print /home/ubuntu/cephtest/cluster1.monmap 2024-04-24T22:27:18.778 INFO:teuthology.orchestra.run.smithi080.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-04-24T22:27:18.779 INFO:teuthology.orchestra.run.smithi080.stdout:monmaptool: generated fsid c6f15796-edf2-4aa3-acbd-00f4afb79b95 2024-04-24T22:27:18.779 INFO:teuthology.orchestra.run.smithi080.stdout:setting min_mon_release = octopus 2024-04-24T22:27:18.779 INFO:teuthology.orchestra.run.smithi080.stdout:epoch 0 2024-04-24T22:27:18.779 INFO:teuthology.orchestra.run.smithi080.stdout:fsid c6f15796-edf2-4aa3-acbd-00f4afb79b95 2024-04-24T22:27:18.779 INFO:teuthology.orchestra.run.smithi080.stdout:last_changed 2024-04-24T22:27:18.778481+0000 2024-04-24T22:27:18.779 INFO:teuthology.orchestra.run.smithi080.stdout:created 2024-04-24T22:27:18.778481+0000 2024-04-24T22:27:18.779 INFO:teuthology.orchestra.run.smithi080.stdout:min_mon_release 15 (octopus) 2024-04-24T22:27:18.779 INFO:teuthology.orchestra.run.smithi080.stdout:election_strategy: 1 2024-04-24T22:27:18.779 INFO:teuthology.orchestra.run.smithi080.stdout:0: [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] mon.a 2024-04-24T22:27:18.779 INFO:teuthology.orchestra.run.smithi080.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-04-24T22:27:18.781 DEBUG:teuthology.orchestra.run.smithi080:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T22:27:18.838 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID c6f15796-edf2-4aa3-acbd-00f4afb79b95... 2024-04-24T22:27:18.839 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-04-24T22:27:18.881 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-04-24T22:27:18.922 INFO:teuthology.orchestra.run.smithi080.stdout:[global] 2024-04-24T22:27:18.923 INFO:teuthology.orchestra.run.smithi080.stdout: chdir = "" 2024-04-24T22:27:18.923 INFO:teuthology.orchestra.run.smithi080.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T22:27:18.923 INFO:teuthology.orchestra.run.smithi080.stdout: auth supported = cephx 2024-04-24T22:27:18.923 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.923 INFO:teuthology.orchestra.run.smithi080.stdout: filestore xattr use omap = true 2024-04-24T22:27:18.923 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.923 INFO:teuthology.orchestra.run.smithi080.stdout: mon clock drift allowed = 1.000 2024-04-24T22:27:18.923 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.924 INFO:teuthology.orchestra.run.smithi080.stdout: osd crush chooseleaf type = 0 2024-04-24T22:27:18.924 INFO:teuthology.orchestra.run.smithi080.stdout: auth debug = true 2024-04-24T22:27:18.924 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.924 INFO:teuthology.orchestra.run.smithi080.stdout: ms die on old message = true 2024-04-24T22:27:18.924 INFO:teuthology.orchestra.run.smithi080.stdout: ms die on bug = true 2024-04-24T22:27:18.924 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.924 INFO:teuthology.orchestra.run.smithi080.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T22:27:18.924 INFO:teuthology.orchestra.run.smithi080.stdout: mon pg warn max object skew = 0 2024-04-24T22:27:18.925 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.925 INFO:teuthology.orchestra.run.smithi080.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T22:27:18.925 INFO:teuthology.orchestra.run.smithi080.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T22:27:18.925 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.925 INFO:teuthology.orchestra.run.smithi080.stdout: osd pool default size = 2 2024-04-24T22:27:18.925 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.925 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd allow primary affinity = true 2024-04-24T22:27:18.925 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd allow pg remap = true 2024-04-24T22:27:18.925 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on legacy crush tunables = false 2024-04-24T22:27:18.925 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on crush straw calc version zero = false 2024-04-24T22:27:18.925 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on no sortbitwise = false 2024-04-24T22:27:18.925 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on osd down out interval zero = false 2024-04-24T22:27:18.925 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on too few osds = false 2024-04-24T22:27:18.925 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T22:27:18.925 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T22:27:18.925 INFO:teuthology.orchestra.run.smithi080.stdout: mon_allow_pool_size_one = true 2024-04-24T22:27:18.926 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.926 INFO:teuthology.orchestra.run.smithi080.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T22:27:18.926 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.926 INFO:teuthology.orchestra.run.smithi080.stdout: osd default data pool replay window = 5 2024-04-24T22:27:18.926 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.926 INFO:teuthology.orchestra.run.smithi080.stdout: mon allow pool delete = true 2024-04-24T22:27:18.926 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.926 INFO:teuthology.orchestra.run.smithi080.stdout: mon cluster log file level = debug 2024-04-24T22:27:18.926 INFO:teuthology.orchestra.run.smithi080.stdout: debug asserts on shutdown = true 2024-04-24T22:27:18.926 INFO:teuthology.orchestra.run.smithi080.stdout: mon health detail to clog = false 2024-04-24T22:27:18.926 INFO:teuthology.orchestra.run.smithi080.stdout: mon host = 172.21.15.80 2024-04-24T22:27:18.926 INFO:teuthology.orchestra.run.smithi080.stdout: mon client directed command retry = 5 2024-04-24T22:27:18.926 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on pool no app = False 2024-04-24T22:27:18.926 INFO:teuthology.orchestra.run.smithi080.stdout: ms inject socket failures = 5000 2024-04-24T22:27:18.927 INFO:teuthology.orchestra.run.smithi080.stdout: fsid = c6f15796-edf2-4aa3-acbd-00f4afb79b95 2024-04-24T22:27:18.927 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.927 INFO:teuthology.orchestra.run.smithi080.stdout:[osd] 2024-04-24T22:27:18.927 INFO:teuthology.orchestra.run.smithi080.stdout: osd journal size = 100 2024-04-24T22:27:18.927 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.927 INFO:teuthology.orchestra.run.smithi080.stdout: osd scrub load threshold = 5.0 2024-04-24T22:27:18.927 INFO:teuthology.orchestra.run.smithi080.stdout: osd scrub max interval = 600 2024-04-24T22:27:18.927 INFO:teuthology.orchestra.run.smithi080.stdout: osd mclock profile = high_recovery_ops 2024-04-24T22:27:18.928 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.928 INFO:teuthology.orchestra.run.smithi080.stdout: osd recover clone overlap = true 2024-04-24T22:27:18.928 INFO:teuthology.orchestra.run.smithi080.stdout: osd recovery max chunk = 1048576 2024-04-24T22:27:18.928 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.928 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug shutdown = true 2024-04-24T22:27:18.928 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug op order = true 2024-04-24T22:27:18.928 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug verify stray on activate = true 2024-04-24T22:27:18.928 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.928 INFO:teuthology.orchestra.run.smithi080.stdout: osd open classes on start = true 2024-04-24T22:27:18.928 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug pg log writeout = true 2024-04-24T22:27:18.928 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.928 INFO:teuthology.orchestra.run.smithi080.stdout: osd deep scrub update digest min age = 30 2024-04-24T22:27:18.928 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.928 INFO:teuthology.orchestra.run.smithi080.stdout: osd map max advance = 10 2024-04-24T22:27:18.929 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.929 INFO:teuthology.orchestra.run.smithi080.stdout: journal zero on create = true 2024-04-24T22:27:18.929 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.929 INFO:teuthology.orchestra.run.smithi080.stdout: filestore ondisk finisher threads = 3 2024-04-24T22:27:18.929 INFO:teuthology.orchestra.run.smithi080.stdout: filestore apply finisher threads = 3 2024-04-24T22:27:18.929 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.929 INFO:teuthology.orchestra.run.smithi080.stdout: bdev debug aio = true 2024-04-24T22:27:18.929 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug misdirected ops = true 2024-04-24T22:27:18.929 INFO:teuthology.orchestra.run.smithi080.stdout: bluestore block size = 96636764160 2024-04-24T22:27:18.929 INFO:teuthology.orchestra.run.smithi080.stdout: bluestore compression algorithm = lz4 2024-04-24T22:27:18.929 INFO:teuthology.orchestra.run.smithi080.stdout: bluestore compression mode = aggressive 2024-04-24T22:27:18.929 INFO:teuthology.orchestra.run.smithi080.stdout: bluestore fsck on mount = True 2024-04-24T22:27:18.929 INFO:teuthology.orchestra.run.smithi080.stdout: debug bluefs = 1/20 2024-04-24T22:27:18.929 INFO:teuthology.orchestra.run.smithi080.stdout: debug bluestore = 1/20 2024-04-24T22:27:18.929 INFO:teuthology.orchestra.run.smithi080.stdout: debug ms = 1 2024-04-24T22:27:18.930 INFO:teuthology.orchestra.run.smithi080.stdout: debug osd = 20 2024-04-24T22:27:18.930 INFO:teuthology.orchestra.run.smithi080.stdout: debug rocksdb = 4/10 2024-04-24T22:27:18.930 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T22:27:18.930 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd full ratio = 0.9 2024-04-24T22:27:18.930 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd nearfull ratio = 0.8 2024-04-24T22:27:18.930 INFO:teuthology.orchestra.run.smithi080.stdout: osd failsafe full ratio = 0.95 2024-04-24T22:27:18.930 INFO:teuthology.orchestra.run.smithi080.stdout: osd objectstore = bluestore 2024-04-24T22:27:18.930 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.930 INFO:teuthology.orchestra.run.smithi080.stdout:[mgr] 2024-04-24T22:27:18.930 INFO:teuthology.orchestra.run.smithi080.stdout: debug ms = 1 2024-04-24T22:27:18.930 INFO:teuthology.orchestra.run.smithi080.stdout: debug mgr = 20 2024-04-24T22:27:18.930 INFO:teuthology.orchestra.run.smithi080.stdout: debug mon = 20 2024-04-24T22:27:18.930 INFO:teuthology.orchestra.run.smithi080.stdout: debug auth = 20 2024-04-24T22:27:18.930 INFO:teuthology.orchestra.run.smithi080.stdout: mon reweight min pgs per osd = 4 2024-04-24T22:27:18.931 INFO:teuthology.orchestra.run.smithi080.stdout: mon reweight min bytes per osd = 10 2024-04-24T22:27:18.931 INFO:teuthology.orchestra.run.smithi080.stdout: mgr/telemetry/nag = false 2024-04-24T22:27:18.931 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.931 INFO:teuthology.orchestra.run.smithi080.stdout:[mon] 2024-04-24T22:27:18.931 INFO:teuthology.orchestra.run.smithi080.stdout: debug ms = 1 2024-04-24T22:27:18.931 INFO:teuthology.orchestra.run.smithi080.stdout: debug mon = 20 2024-04-24T22:27:18.931 INFO:teuthology.orchestra.run.smithi080.stdout: debug paxos = 20 2024-04-24T22:27:18.931 INFO:teuthology.orchestra.run.smithi080.stdout: debug auth = 20 2024-04-24T22:27:18.931 INFO:teuthology.orchestra.run.smithi080.stdout: mon data avail warn = 5 2024-04-24T22:27:18.931 INFO:teuthology.orchestra.run.smithi080.stdout: mon mgr mkfs grace = 240 2024-04-24T22:27:18.931 INFO:teuthology.orchestra.run.smithi080.stdout: mon reweight min pgs per osd = 4 2024-04-24T22:27:18.931 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd reporter subtree level = osd 2024-04-24T22:27:18.931 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd prime pg temp = true 2024-04-24T22:27:18.932 INFO:teuthology.orchestra.run.smithi080.stdout: mon reweight min bytes per osd = 10 2024-04-24T22:27:18.932 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.932 INFO:teuthology.orchestra.run.smithi080.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T22:27:18.932 INFO:teuthology.orchestra.run.smithi080.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T22:27:18.932 INFO:teuthology.orchestra.run.smithi080.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T22:27:18.932 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.932 INFO:teuthology.orchestra.run.smithi080.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T22:27:18.932 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T22:27:18.932 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T22:27:18.932 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.932 INFO:teuthology.orchestra.run.smithi080.stdout: # 1m isn't quite enough 2024-04-24T22:27:18.932 INFO:teuthology.orchestra.run.smithi080.stdout: mon_down_mkfs_grace = 2m 2024-04-24T22:27:18.933 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.933 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_filestore_osds = false 2024-04-24T22:27:18.933 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:18.933 INFO:teuthology.orchestra.run.smithi080.stdout:[client] 2024-04-24T22:27:18.933 INFO:teuthology.orchestra.run.smithi080.stdout: rgw cache enabled = true 2024-04-24T22:27:18.933 INFO:teuthology.orchestra.run.smithi080.stdout: rgw enable ops log = true 2024-04-24T22:27:18.933 INFO:teuthology.orchestra.run.smithi080.stdout: rgw enable usage log = true 2024-04-24T22:27:18.933 INFO:teuthology.orchestra.run.smithi080.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T22:27:18.933 INFO:teuthology.orchestra.run.smithi080.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T22:27:18.933 INFO:teuthology.orchestra.run.smithi080.stdout: debug rbd = 20 2024-04-24T22:27:18.933 INFO:teuthology.orchestra.run.smithi080.stdout: debug rbd_mirror = 15 2024-04-24T22:27:18.934 INFO:teuthology.orchestra.run.smithi080.stdout: log to stderr = False 2024-04-24T22:27:18.934 INFO:teuthology.orchestra.run.smithi080.stdout: rbd default features = 125 2024-04-24T22:27:18.934 INFO:teuthology.orchestra.run.smithi080.stdout: rbd mirror image policy type = none 2024-04-24T22:27:18.934 INFO:teuthology.orchestra.run.smithi080.stdout:[mon.a] 2024-04-24T22:27:18.934 INFO:teuthology.orchestra.run.smithi080.stdout:[client.mirror.0] 2024-04-24T22:27:18.934 INFO:teuthology.orchestra.run.smithi080.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:18.934 INFO:teuthology.orchestra.run.smithi080.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:18.935 INFO:teuthology.orchestra.run.smithi080.stdout:[client.mirror.1] 2024-04-24T22:27:18.935 INFO:teuthology.orchestra.run.smithi080.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:18.935 INFO:teuthology.orchestra.run.smithi080.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:18.935 INFO:teuthology.orchestra.run.smithi080.stdout:[client.mirror.2] 2024-04-24T22:27:18.935 INFO:teuthology.orchestra.run.smithi080.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:18.935 INFO:teuthology.orchestra.run.smithi080.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:18.935 INFO:teuthology.orchestra.run.smithi080.stdout:[client.mirror.3] 2024-04-24T22:27:18.935 INFO:teuthology.orchestra.run.smithi080.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:18.935 INFO:teuthology.orchestra.run.smithi080.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:18.936 INFO:teuthology.orchestra.run.smithi134.stdout:[global] 2024-04-24T22:27:18.936 INFO:teuthology.orchestra.run.smithi134.stdout: chdir = "" 2024-04-24T22:27:18.936 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T22:27:18.936 INFO:teuthology.orchestra.run.smithi134.stdout: auth supported = cephx 2024-04-24T22:27:18.936 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.937 INFO:teuthology.orchestra.run.smithi134.stdout: filestore xattr use omap = true 2024-04-24T22:27:18.937 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.937 INFO:teuthology.orchestra.run.smithi134.stdout: mon clock drift allowed = 1.000 2024-04-24T22:27:18.937 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.937 INFO:teuthology.orchestra.run.smithi134.stdout: osd crush chooseleaf type = 0 2024-04-24T22:27:18.937 INFO:teuthology.orchestra.run.smithi134.stdout: auth debug = true 2024-04-24T22:27:18.937 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.937 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on old message = true 2024-04-24T22:27:18.937 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on bug = true 2024-04-24T22:27:18.937 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.937 INFO:teuthology.orchestra.run.smithi134.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T22:27:18.937 INFO:teuthology.orchestra.run.smithi134.stdout: mon pg warn max object skew = 0 2024-04-24T22:27:18.937 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.938 INFO:teuthology.orchestra.run.smithi134.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T22:27:18.938 INFO:teuthology.orchestra.run.smithi134.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T22:27:18.938 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.938 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default size = 2 2024-04-24T22:27:18.938 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.938 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow primary affinity = true 2024-04-24T22:27:18.939 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow pg remap = true 2024-04-24T22:27:18.939 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on legacy crush tunables = false 2024-04-24T22:27:18.939 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on crush straw calc version zero = false 2024-04-24T22:27:18.939 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on no sortbitwise = false 2024-04-24T22:27:18.939 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on osd down out interval zero = false 2024-04-24T22:27:18.939 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on too few osds = false 2024-04-24T22:27:18.939 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T22:27:18.939 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T22:27:18.939 INFO:teuthology.orchestra.run.smithi134.stdout: mon_allow_pool_size_one = true 2024-04-24T22:27:18.939 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.939 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T22:27:18.939 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.939 INFO:teuthology.orchestra.run.smithi134.stdout: osd default data pool replay window = 5 2024-04-24T22:27:18.939 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.939 INFO:teuthology.orchestra.run.smithi134.stdout: mon allow pool delete = true 2024-04-24T22:27:18.940 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.940 INFO:teuthology.orchestra.run.smithi134.stdout: mon cluster log file level = debug 2024-04-24T22:27:18.940 INFO:teuthology.orchestra.run.smithi134.stdout: debug asserts on shutdown = true 2024-04-24T22:27:18.940 INFO:teuthology.orchestra.run.smithi134.stdout: mon health detail to clog = false 2024-04-24T22:27:18.940 INFO:teuthology.orchestra.run.smithi134.stdout: mon host = 172.21.15.80 2024-04-24T22:27:18.940 INFO:teuthology.orchestra.run.smithi134.stdout: mon client directed command retry = 5 2024-04-24T22:27:18.940 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on pool no app = False 2024-04-24T22:27:18.940 INFO:teuthology.orchestra.run.smithi134.stdout: ms inject socket failures = 5000 2024-04-24T22:27:18.940 INFO:teuthology.orchestra.run.smithi134.stdout: fsid = c6f15796-edf2-4aa3-acbd-00f4afb79b95 2024-04-24T22:27:18.940 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.940 INFO:teuthology.orchestra.run.smithi134.stdout:[osd] 2024-04-24T22:27:18.940 INFO:teuthology.orchestra.run.smithi134.stdout: osd journal size = 100 2024-04-24T22:27:18.940 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.940 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub load threshold = 5.0 2024-04-24T22:27:18.940 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub max interval = 600 2024-04-24T22:27:18.941 INFO:teuthology.orchestra.run.smithi134.stdout: osd mclock profile = high_recovery_ops 2024-04-24T22:27:18.941 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.941 INFO:teuthology.orchestra.run.smithi134.stdout: osd recover clone overlap = true 2024-04-24T22:27:18.941 INFO:teuthology.orchestra.run.smithi134.stdout: osd recovery max chunk = 1048576 2024-04-24T22:27:18.941 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.941 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug shutdown = true 2024-04-24T22:27:18.941 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug op order = true 2024-04-24T22:27:18.941 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug verify stray on activate = true 2024-04-24T22:27:18.941 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.942 INFO:teuthology.orchestra.run.smithi134.stdout: osd open classes on start = true 2024-04-24T22:27:18.942 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug pg log writeout = true 2024-04-24T22:27:18.942 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.942 INFO:teuthology.orchestra.run.smithi134.stdout: osd deep scrub update digest min age = 30 2024-04-24T22:27:18.942 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.942 INFO:teuthology.orchestra.run.smithi134.stdout: osd map max advance = 10 2024-04-24T22:27:18.942 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.942 INFO:teuthology.orchestra.run.smithi134.stdout: journal zero on create = true 2024-04-24T22:27:18.942 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.942 INFO:teuthology.orchestra.run.smithi134.stdout: filestore ondisk finisher threads = 3 2024-04-24T22:27:18.942 INFO:teuthology.orchestra.run.smithi134.stdout: filestore apply finisher threads = 3 2024-04-24T22:27:18.943 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.943 INFO:teuthology.orchestra.run.smithi134.stdout: bdev debug aio = true 2024-04-24T22:27:18.943 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug misdirected ops = true 2024-04-24T22:27:18.943 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore block size = 96636764160 2024-04-24T22:27:18.943 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore compression algorithm = lz4 2024-04-24T22:27:18.943 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore compression mode = aggressive 2024-04-24T22:27:18.943 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore fsck on mount = True 2024-04-24T22:27:18.943 INFO:teuthology.orchestra.run.smithi134.stdout: debug bluefs = 1/20 2024-04-24T22:27:18.943 INFO:teuthology.orchestra.run.smithi134.stdout: debug bluestore = 1/20 2024-04-24T22:27:18.943 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-04-24T22:27:18.943 INFO:teuthology.orchestra.run.smithi134.stdout: debug osd = 20 2024-04-24T22:27:18.944 INFO:teuthology.orchestra.run.smithi134.stdout: debug rocksdb = 4/10 2024-04-24T22:27:18.944 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T22:27:18.944 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd full ratio = 0.9 2024-04-24T22:27:18.944 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd nearfull ratio = 0.8 2024-04-24T22:27:18.944 INFO:teuthology.orchestra.run.smithi134.stdout: osd failsafe full ratio = 0.95 2024-04-24T22:27:18.944 INFO:teuthology.orchestra.run.smithi134.stdout: osd objectstore = bluestore 2024-04-24T22:27:18.944 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.944 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr] 2024-04-24T22:27:18.944 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-04-24T22:27:18.944 INFO:teuthology.orchestra.run.smithi134.stdout: debug mgr = 20 2024-04-24T22:27:18.945 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-04-24T22:27:18.945 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-04-24T22:27:18.945 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-04-24T22:27:18.945 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-04-24T22:27:18.945 INFO:teuthology.orchestra.run.smithi134.stdout: mgr/telemetry/nag = false 2024-04-24T22:27:18.945 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.945 INFO:teuthology.orchestra.run.smithi134.stdout:[mon] 2024-04-24T22:27:18.945 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-04-24T22:27:18.945 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-04-24T22:27:18.945 INFO:teuthology.orchestra.run.smithi134.stdout: debug paxos = 20 2024-04-24T22:27:18.946 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-04-24T22:27:18.946 INFO:teuthology.orchestra.run.smithi134.stdout: mon data avail warn = 5 2024-04-24T22:27:18.946 INFO:teuthology.orchestra.run.smithi134.stdout: mon mgr mkfs grace = 240 2024-04-24T22:27:18.946 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-04-24T22:27:18.946 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd reporter subtree level = osd 2024-04-24T22:27:18.946 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd prime pg temp = true 2024-04-24T22:27:18.946 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-04-24T22:27:18.946 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.946 INFO:teuthology.orchestra.run.smithi134.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T22:27:18.946 INFO:teuthology.orchestra.run.smithi134.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T22:27:18.947 INFO:teuthology.orchestra.run.smithi134.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T22:27:18.947 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.947 INFO:teuthology.orchestra.run.smithi134.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T22:27:18.947 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T22:27:18.947 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T22:27:18.947 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.947 INFO:teuthology.orchestra.run.smithi134.stdout: # 1m isn't quite enough 2024-04-24T22:27:18.948 INFO:teuthology.orchestra.run.smithi134.stdout: mon_down_mkfs_grace = 2m 2024-04-24T22:27:18.948 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.948 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_filestore_osds = false 2024-04-24T22:27:18.948 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:18.948 INFO:teuthology.orchestra.run.smithi134.stdout:[client] 2024-04-24T22:27:18.948 INFO:teuthology.orchestra.run.smithi134.stdout: rgw cache enabled = true 2024-04-24T22:27:18.949 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable ops log = true 2024-04-24T22:27:18.949 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable usage log = true 2024-04-24T22:27:18.949 INFO:teuthology.orchestra.run.smithi134.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T22:27:18.949 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T22:27:18.949 INFO:teuthology.orchestra.run.smithi134.stdout: debug rbd = 20 2024-04-24T22:27:18.949 INFO:teuthology.orchestra.run.smithi134.stdout: debug rbd_mirror = 15 2024-04-24T22:27:18.950 INFO:teuthology.orchestra.run.smithi134.stdout: log to stderr = False 2024-04-24T22:27:18.950 INFO:teuthology.orchestra.run.smithi134.stdout: rbd default features = 125 2024-04-24T22:27:18.950 INFO:teuthology.orchestra.run.smithi134.stdout: rbd mirror image policy type = none 2024-04-24T22:27:18.950 INFO:teuthology.orchestra.run.smithi134.stdout:[mon.a] 2024-04-24T22:27:18.950 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.0] 2024-04-24T22:27:18.950 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:18.950 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:18.951 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.1] 2024-04-24T22:27:18.951 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:18.951 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:18.951 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.2] 2024-04-24T22:27:18.951 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:18.951 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:18.951 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.3] 2024-04-24T22:27:18.952 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:18.952 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:18.954 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-04-24T22:27:18.954 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-04-24T22:27:19.047 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-24T22:27:19.047 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:27:19.047 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-04-24T22:27:19.065 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:27:19.065 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-04-24T22:27:19.123 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi080.front.sepia.ceph.com 2024-04-24T22:27:19.123 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:27:19.123 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/ceph/cluster1.keyring 2024-04-24T22:27:19.123 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-04-24T22:27:19.200 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:27:19.200 DEBUG:teuthology.orchestra.run.smithi080:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-04-24T22:27:19.258 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi134.front.sepia.ceph.com 2024-04-24T22:27:19.258 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:27:19.258 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/cluster1.keyring 2024-04-24T22:27:19.258 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-04-24T22:27:19.302 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:27:19.302 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-04-24T22:27:19.363 INFO:tasks.ceph:Setting up mon nodes... 2024-04-24T22:27:19.363 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-24T22:27:19.363 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-04-24T22:27:19.413 INFO:teuthology.orchestra.run.smithi080.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-04-24T22:27:19.415 INFO:tasks.ceph:Setting up mds nodes... 2024-04-24T22:27:19.415 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-24T22:27:19.416 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-04-24T22:27:19.457 INFO:teuthology.orchestra.run.smithi080.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-04-24T22:27:19.467 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-04-24T22:27:19.526 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-04-24T22:27:19.540 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-04-24T22:27:19.583 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-04-24T22:27:19.595 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-04-24T22:27:19.679 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-04-24T22:27:19.691 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-04-24T22:27:19.772 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-04-24T22:27:19.784 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-04-24T22:27:19.864 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-04-24T22:27:19.876 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-04-24T22:27:19.956 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-04-24T22:27:19.968 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-04-24T22:27:20.048 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-04-24T22:27:20.059 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-04-24T22:27:20.140 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-04-24T22:27:20.153 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-24T22:27:20.153 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi080.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-24T22:27:20.153 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-04-24T22:27:20.184 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T22:27:20.184 INFO:tasks.ceph:role: cluster1.osd.0 2024-04-24T22:27:20.184 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi080.front.sepia.ceph.com 2024-04-24T22:27:20.184 DEBUG:teuthology.orchestra.run.smithi080:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-24T22:27:20.263 INFO:teuthology.orchestra.run.smithi080.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T22:27:20.263 INFO:teuthology.orchestra.run.smithi080.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T22:27:20.263 INFO:teuthology.orchestra.run.smithi080.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T22:27:20.263 INFO:teuthology.orchestra.run.smithi080.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T22:27:20.263 INFO:teuthology.orchestra.run.smithi080.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T22:27:20.263 INFO:teuthology.orchestra.run.smithi080.stdout: = sunit=0 swidth=0 blks 2024-04-24T22:27:20.263 INFO:teuthology.orchestra.run.smithi080.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T22:27:20.264 INFO:teuthology.orchestra.run.smithi080.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T22:27:20.264 INFO:teuthology.orchestra.run.smithi080.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T22:27:20.264 INFO:teuthology.orchestra.run.smithi080.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T22:27:20.696 INFO:teuthology.orchestra.run.smithi080.stdout:Discarding blocks...Done. 2024-04-24T22:27:20.698 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi080.front.sepia.ceph.com -o noatime 2024-04-24T22:27:20.698 DEBUG:teuthology.orchestra.run.smithi080:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-04-24T22:27:20.864 DEBUG:teuthology.orchestra.run.smithi080:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-04-24T22:27:20.900 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-04-24T22:27:20.969 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T22:27:20.969 INFO:tasks.ceph:role: cluster1.osd.1 2024-04-24T22:27:20.969 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi080.front.sepia.ceph.com 2024-04-24T22:27:20.969 DEBUG:teuthology.orchestra.run.smithi080:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-24T22:27:21.044 INFO:teuthology.orchestra.run.smithi080.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T22:27:21.044 INFO:teuthology.orchestra.run.smithi080.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T22:27:21.045 INFO:teuthology.orchestra.run.smithi080.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T22:27:21.045 INFO:teuthology.orchestra.run.smithi080.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T22:27:21.045 INFO:teuthology.orchestra.run.smithi080.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T22:27:21.045 INFO:teuthology.orchestra.run.smithi080.stdout: = sunit=0 swidth=0 blks 2024-04-24T22:27:21.045 INFO:teuthology.orchestra.run.smithi080.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T22:27:21.045 INFO:teuthology.orchestra.run.smithi080.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T22:27:21.045 INFO:teuthology.orchestra.run.smithi080.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T22:27:21.045 INFO:teuthology.orchestra.run.smithi080.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T22:27:21.472 INFO:teuthology.orchestra.run.smithi080.stdout:Discarding blocks...Done. 2024-04-24T22:27:21.474 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi080.front.sepia.ceph.com -o noatime 2024-04-24T22:27:21.474 DEBUG:teuthology.orchestra.run.smithi080:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-04-24T22:27:21.583 DEBUG:teuthology.orchestra.run.smithi080:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-04-24T22:27:21.615 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-04-24T22:27:21.683 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T22:27:21.684 INFO:tasks.ceph:role: cluster1.osd.2 2024-04-24T22:27:21.684 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi080.front.sepia.ceph.com 2024-04-24T22:27:21.684 DEBUG:teuthology.orchestra.run.smithi080:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-24T22:27:21.758 INFO:teuthology.orchestra.run.smithi080.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T22:27:21.758 INFO:teuthology.orchestra.run.smithi080.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T22:27:21.758 INFO:teuthology.orchestra.run.smithi080.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T22:27:21.758 INFO:teuthology.orchestra.run.smithi080.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T22:27:21.758 INFO:teuthology.orchestra.run.smithi080.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T22:27:21.758 INFO:teuthology.orchestra.run.smithi080.stdout: = sunit=0 swidth=0 blks 2024-04-24T22:27:21.758 INFO:teuthology.orchestra.run.smithi080.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T22:27:21.758 INFO:teuthology.orchestra.run.smithi080.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T22:27:21.758 INFO:teuthology.orchestra.run.smithi080.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T22:27:21.759 INFO:teuthology.orchestra.run.smithi080.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T22:27:22.204 INFO:teuthology.orchestra.run.smithi080.stdout:Discarding blocks...Done. 2024-04-24T22:27:22.206 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi080.front.sepia.ceph.com -o noatime 2024-04-24T22:27:22.210 DEBUG:teuthology.orchestra.run.smithi080:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-04-24T22:27:22.352 DEBUG:teuthology.orchestra.run.smithi080:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-04-24T22:27:22.385 DEBUG:teuthology.orchestra.run.smithi080:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-04-24T22:27:22.478 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:22.477+0000 7f9ec8e82540 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-04-24T22:27:22.480 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:22.479+0000 7f9ec8e82540 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-04-24T22:27:22.480 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:22.479+0000 7f9ec8e82540 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-04-24T22:27:22.480 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:22.479+0000 7f9ec8e82540 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-04-24T22:27:22.481 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:22.479+0000 7f9ec8e82540 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-04-24T22:27:22.485 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:22.484+0000 7f9ec8e82540 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-04-24T22:27:23.010 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:23.008+0000 7f9ec8e82540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T22:27:23.774 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:23.773+0000 7f9ec8e82540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T22:27:24.538 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:24.536+0000 7f9ec8e82540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T22:27:25.334 DEBUG:teuthology.orchestra.run.smithi080:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-04-24T22:27:25.363 DEBUG:teuthology.orchestra.run.smithi080:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-04-24T22:27:25.448 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:25.446+0000 7fd6269c9540 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-04-24T22:27:25.448 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:25.446+0000 7fd6269c9540 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-04-24T22:27:25.448 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:25.446+0000 7fd6269c9540 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-04-24T22:27:25.448 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:25.446+0000 7fd6269c9540 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-04-24T22:27:25.448 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:25.446+0000 7fd6269c9540 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-04-24T22:27:25.448 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:25.446+0000 7fd6269c9540 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-04-24T22:27:25.972 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:25.970+0000 7fd6269c9540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T22:27:26.737 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:26.735+0000 7fd6269c9540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T22:27:27.500 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:27.498+0000 7fd6269c9540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T22:27:28.291 DEBUG:teuthology.orchestra.run.smithi080:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-04-24T22:27:28.323 DEBUG:teuthology.orchestra.run.smithi080:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-04-24T22:27:28.414 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:28.411+0000 7f9dde2ca540 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-04-24T22:27:28.414 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:28.412+0000 7f9dde2ca540 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-04-24T22:27:28.414 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:28.412+0000 7f9dde2ca540 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-04-24T22:27:28.414 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:28.412+0000 7f9dde2ca540 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-04-24T22:27:28.414 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:28.412+0000 7f9dde2ca540 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-04-24T22:27:28.414 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:28.412+0000 7f9dde2ca540 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-04-24T22:27:28.938 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:28.937+0000 7f9dde2ca540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T22:27:29.707 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:29.705+0000 7f9dde2ca540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T22:27:30.475 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T22:27:30.473+0000 7f9dde2ca540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T22:27:31.265 DEBUG:teuthology.orchestra.run.smithi080:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-04-24T22:27:31.298 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-24T22:27:31.298 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:27:31.298 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-04-24T22:27:31.371 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:27:31.371 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-04-24T22:27:31.443 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:27:31.444 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-04-24T22:27:31.516 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:27:31.516 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-04-24T22:27:31.588 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:27:31.588 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-04-24T22:27:31.650 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:27:31.650 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-04-24T22:27:31.672 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:27:31.673 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-04-24T22:27:31.733 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:27:31.733 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-04-24T22:27:31.793 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:27:31.793 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-04-24T22:27:31.853 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:27:31.853 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-04-24T22:27:31.912 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:27:31.912 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-04-24T22:27:31.971 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:27:31.971 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-04-24T22:27:32.030 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:27:32.030 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-04-24T22:27:32.088 INFO:tasks.ceph:Adding keys to all mons... 2024-04-24T22:27:32.089 DEBUG:teuthology.orchestra.run.smithi080:> sudo tee -a /etc/ceph/cluster1.keyring 2024-04-24T22:27:32.118 INFO:teuthology.orchestra.run.smithi080.stdout:[mgr.x] 2024-04-24T22:27:32.118 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQBHhylmK66gGBAAtILDQs9bVnEqkr6u6sqmUg== 2024-04-24T22:27:32.118 INFO:teuthology.orchestra.run.smithi080.stdout:[osd.0] 2024-04-24T22:27:32.118 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQBKhylmgjSDHBAA4dqc4e1K20HWWPOaZllMjg== 2024-04-24T22:27:32.118 INFO:teuthology.orchestra.run.smithi080.stdout:[osd.1] 2024-04-24T22:27:32.118 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQBNhylmp9qqGhAAZQooxRKV1P6L5EOTMnS4MA== 2024-04-24T22:27:32.118 INFO:teuthology.orchestra.run.smithi080.stdout:[osd.2] 2024-04-24T22:27:32.118 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQBQhylm1GufGBAA495rxD1kSpHAg6mVVNUPtA== 2024-04-24T22:27:32.119 INFO:teuthology.orchestra.run.smithi080.stdout:[client.0] 2024-04-24T22:27:32.119 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQBHhylmCgwmGxAAKC1rLpfuz8kEko414/z4bQ== 2024-04-24T22:27:32.119 INFO:teuthology.orchestra.run.smithi080.stdout:[client.mirror] 2024-04-24T22:27:32.119 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQBHhylmeeVbHxAAiLXEOKCgOZ9y7fy1Nr8n5Q== 2024-04-24T22:27:32.119 INFO:teuthology.orchestra.run.smithi080.stdout:[client.mirror.0] 2024-04-24T22:27:32.119 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQBHhylm+P69IhAAB6IM5S9V21iKN6LzgtzqfQ== 2024-04-24T22:27:32.119 INFO:teuthology.orchestra.run.smithi080.stdout:[client.mirror.1] 2024-04-24T22:27:32.119 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQBHhylma3R7KBAAz1fwZ3xDxPiMgjTE+0iQXw== 2024-04-24T22:27:32.119 INFO:teuthology.orchestra.run.smithi080.stdout:[client.mirror.2] 2024-04-24T22:27:32.119 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQBHhylm+FYFLhAANBVhP9w0cu9WrB2PeX/ANA== 2024-04-24T22:27:32.119 INFO:teuthology.orchestra.run.smithi080.stdout:[client.mirror.3] 2024-04-24T22:27:32.119 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQBHhylmUsuBMxAAR9NNLsMRmN0+O5Y8ITTu8Q== 2024-04-24T22:27:32.119 INFO:teuthology.orchestra.run.smithi080.stdout:[client.mirror.4] 2024-04-24T22:27:32.119 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQBHhylmm7H8OBAAexYUZuTIAF+BiBH2EgF7zA== 2024-04-24T22:27:32.119 INFO:teuthology.orchestra.run.smithi080.stdout:[client.mirror.5] 2024-04-24T22:27:32.120 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQBIhylmWTvYAhAAs/PYeIydJHHSoIY8Y/LfNA== 2024-04-24T22:27:32.120 INFO:teuthology.orchestra.run.smithi080.stdout:[client.mirror.6] 2024-04-24T22:27:32.120 INFO:teuthology.orchestra.run.smithi080.stdout: key = AQBIhylmfZpgCBAAGYGyAABF0/BcfaCsMkU1aQ== 2024-04-24T22:27:32.121 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T22:27:32.214 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T22:27:32.265 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T22:27:32.315 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T22:27:32.365 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T22:27:32.417 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T22:27:32.469 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T22:27:32.522 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T22:27:32.575 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T22:27:32.628 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T22:27:32.680 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T22:27:32.733 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T22:27:32.786 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T22:27:32.838 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-24T22:27:32.838 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-04-24T22:27:32.865 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-04-24T22:27:32.957 DEBUG:teuthology.orchestra.run.smithi080:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-04-24T22:27:32.983 DEBUG:teuthology.orchestra.run.smithi080:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-04-24T22:27:33.041 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-04-24T22:27:33.041 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-04-24T22:27:33.041 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-04-24T22:27:33.084 INFO:tasks.ceph.cluster1.mon.a:Started 2024-04-24T22:27:33.084 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-04-24T22:27:33.084 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-04-24T22:27:33.084 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-04-24T22:27:33.086 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-04-24T22:27:33.086 INFO:tasks.ceph:Setting crush tunables to default 2024-04-24T22:27:33.086 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph --cluster cluster1 osd crush tunables default 2024-04-24T22:27:33.287 INFO:teuthology.orchestra.run.smithi080.stderr:adjusted tunables profile to default 2024-04-24T22:27:33.297 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-04-24T22:27:33.297 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:27:33.297 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-04-24T22:27:33.324 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:27:33.324 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-04-24T22:27:33.390 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:27:33.390 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-04-24T22:27:33.458 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph --cluster cluster1 osd new 30b54047-4212-4616-bdcb-7ed629fbd802 0 2024-04-24T22:27:33.478 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:33.475+0000 7f6c88ebd1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T22:27:33.636 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:33.634+0000 7f6c88ebd1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T22:27:33.729 INFO:teuthology.orchestra.run.smithi080.stdout:0 2024-04-24T22:27:33.742 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph --cluster cluster1 osd new 3137d80f-296e-4fd9-a77f-97031fbe79b2 1 2024-04-24T22:27:33.949 INFO:teuthology.orchestra.run.smithi080.stdout:1 2024-04-24T22:27:33.962 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph --cluster cluster1 osd new f5cd2fbf-7851-4596-bdee-30b92841e8e4 2 2024-04-24T22:27:34.000 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:33.998+0000 7f6c88ebd1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T22:27:34.170 INFO:teuthology.orchestra.run.smithi080.stdout:2 2024-04-24T22:27:34.183 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-04-24T22:27:34.183 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-04-24T22:27:34.225 INFO:tasks.ceph.cluster1.osd.0:Started 2024-04-24T22:27:34.225 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-04-24T22:27:34.225 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-04-24T22:27:34.227 INFO:tasks.ceph.cluster1.osd.1:Started 2024-04-24T22:27:34.227 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-04-24T22:27:34.227 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-04-24T22:27:34.229 INFO:tasks.ceph.cluster1.osd.2:Started 2024-04-24T22:27:34.229 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-24T22:27:34.451 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:34.451 INFO:teuthology.orchestra.run.smithi080.stdout:{"epoch":5,"fsid":"c6f15796-edf2-4aa3-acbd-00f4afb79b95","created":"2024-04-24T22:27:33.193401+0000","modified":"2024-04-24T22:27:34.169017+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T22:27:34.169017+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"30b54047-4212-4616-bdcb-7ed629fbd802","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3137d80f-296e-4fd9-a77f-97031fbe79b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f5cd2fbf-7851-4596-bdee-30b92841e8e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T22:27:34.463 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-04-24T22:27:34.463 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-24T22:27:34.527 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:34.525+0000 7f6c88ebd1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T22:27:34.665 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:34.664+0000 7f6c88ebd1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T22:27:34.765 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-04-24T22:27:34.824 INFO:tasks.ceph.cluster1.osd.2.smithi080.stderr:2024-04-24T22:27:34.823+0000 7fa2d873d540 -1 Falling back to public interface 2024-04-24T22:27:34.829 INFO:tasks.ceph.cluster1.osd.1.smithi080.stderr:2024-04-24T22:27:34.827+0000 7f70c3485540 -1 Falling back to public interface 2024-04-24T22:27:34.847 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:27:34.845+0000 7fd67df18540 -1 Falling back to public interface 2024-04-24T22:27:34.955 INFO:teuthology.misc.health.smithi080.stdout: 2024-04-24T22:27:34.956 INFO:teuthology.misc.health.smithi080.stdout:{"epoch":5,"fsid":"c6f15796-edf2-4aa3-acbd-00f4afb79b95","created":"2024-04-24T22:27:33.193401+0000","modified":"2024-04-24T22:27:34.169017+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T22:27:34.169017+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"30b54047-4212-4616-bdcb-7ed629fbd802","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3137d80f-296e-4fd9-a77f-97031fbe79b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f5cd2fbf-7851-4596-bdee-30b92841e8e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T22:27:34.966 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-24T22:27:35.018 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:35.016+0000 7f6c88ebd1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T22:27:35.748 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:35.746+0000 7f6c88ebd1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T22:27:35.818 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:35.816+0000 7f6c88ebd1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T22:27:35.950 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:35.948+0000 7f6c88ebd1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T22:27:36.017 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:36.015+0000 7f6c88ebd1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T22:27:36.194 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:36.192+0000 7f6c88ebd1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T22:27:36.270 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:36.268+0000 7f6c88ebd1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T22:27:36.344 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:36.342+0000 7f6c88ebd1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T22:27:36.435 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:36.434+0000 7f6c88ebd1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T22:27:36.614 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:36.612+0000 7f6c88ebd1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T22:27:36.693 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:36.690+0000 7f6c88ebd1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T22:27:36.975 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:36.974+0000 7f6c88ebd1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T22:27:37.046 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:37.044+0000 7f6c88ebd1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T22:27:37.551 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:37.549+0000 7f6c88ebd1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T22:27:37.897 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:37.895+0000 7f6c88ebd1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T22:27:38.035 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:38.033+0000 7f6c88ebd1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T22:27:38.322 INFO:tasks.ceph.cluster1.osd.2.smithi080.stderr:2024-04-24T22:27:38.321+0000 7fa2d873d540 -1 osd.2 0 log_to_monitors true 2024-04-24T22:27:38.344 INFO:tasks.ceph.cluster1.osd.1.smithi080.stderr:2024-04-24T22:27:38.343+0000 7f70c3485540 -1 osd.1 0 log_to_monitors true 2024-04-24T22:27:38.355 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:27:38.354+0000 7fd67df18540 -1 osd.0 0 log_to_monitors true 2024-04-24T22:27:38.413 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:38.411+0000 7f6c88ebd1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T22:27:38.555 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:38.553+0000 7f6c88ebd1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T22:27:38.703 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:38.702+0000 7f6c88ebd1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T22:27:38.806 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:38.804+0000 7f6c88ebd1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T22:27:39.830 INFO:tasks.ceph.cluster1.mgr.x.smithi080.stderr:2024-04-24T22:27:39.829+0000 7f6c4d5c1700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-24T22:27:40.270 INFO:tasks.ceph.cluster1.osd.1.smithi080.stderr:2024-04-24T22:27:40.268+0000 7f70ba08a700 -1 osd.1 0 waiting for initial osdmap 2024-04-24T22:27:40.279 INFO:tasks.ceph.cluster1.osd.1.smithi080.stderr:2024-04-24T22:27:40.272+0000 7f70b2e81700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T22:27:40.279 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:27:40.273+0000 7fd674b1d700 -1 osd.0 0 waiting for initial osdmap 2024-04-24T22:27:40.281 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:27:40.276+0000 7fd66d914700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T22:27:40.282 INFO:tasks.ceph.cluster1.osd.2.smithi080.stderr:2024-04-24T22:27:40.280+0000 7fa2cf342700 -1 osd.2 0 waiting for initial osdmap 2024-04-24T22:27:40.285 INFO:tasks.ceph.cluster1.osd.2.smithi080.stderr:2024-04-24T22:27:40.283+0000 7fa2c8139700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T22:27:41.270 DEBUG:teuthology.orchestra.run.smithi080:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-04-24T22:27:41.586 INFO:teuthology.misc.health.smithi080.stdout: 2024-04-24T22:27:41.587 INFO:teuthology.misc.health.smithi080.stdout:{"epoch":8,"fsid":"c6f15796-edf2-4aa3-acbd-00f4afb79b95","created":"2024-04-24T22:27:33.193401+0000","modified":"2024-04-24T22:27:41.275739+0000","last_up_change":"2024-04-24T22:27:41.275739+0000","last_in_change":"2024-04-24T22:27:34.169017+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"30b54047-4212-4616-bdcb-7ed629fbd802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":42679},{"type":"v1","addr":"172.21.15.80:6817","nonce":42679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":42679},{"type":"v1","addr":"172.21.15.80:6819","nonce":42679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":42679},{"type":"v1","addr":"172.21.15.80:6823","nonce":42679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":42679},{"type":"v1","addr":"172.21.15.80:6821","nonce":42679}]},"public_addr":"172.21.15.80:6817/42679","cluster_addr":"172.21.15.80:6819/42679","heartbeat_back_addr":"172.21.15.80:6823/42679","heartbeat_front_addr":"172.21.15.80:6821/42679","state":["exists","up"]},{"osd":1,"uuid":"3137d80f-296e-4fd9-a77f-97031fbe79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":42674},{"type":"v1","addr":"172.21.15.80:6809","nonce":42674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":42674},{"type":"v1","addr":"172.21.15.80:6811","nonce":42674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":42674},{"type":"v1","addr":"172.21.15.80:6815","nonce":42674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":42674},{"type":"v1","addr":"172.21.15.80:6813","nonce":42674}]},"public_addr":"172.21.15.80:6809/42674","cluster_addr":"172.21.15.80:6811/42674","heartbeat_back_addr":"172.21.15.80:6815/42674","heartbeat_front_addr":"172.21.15.80:6813/42674","state":["exists","up"]},{"osd":2,"uuid":"f5cd2fbf-7851-4596-bdee-30b92841e8e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":42673},{"type":"v1","addr":"172.21.15.80:6801","nonce":42673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":42673},{"type":"v1","addr":"172.21.15.80:6803","nonce":42673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":42673},{"type":"v1","addr":"172.21.15.80:6807","nonce":42673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":42673},{"type":"v1","addr":"172.21.15.80:6805","nonce":42673}]},"public_addr":"172.21.15.80:6801/42673","cluster_addr":"172.21.15.80:6803/42673","heartbeat_back_addr":"172.21.15.80:6807/42673","heartbeat_front_addr":"172.21.15.80:6805/42673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T22:27:41.599 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-24T22:27:41.599 INFO:tasks.ceph:Creating RBD pool 2024-04-24T22:27:41.599 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-04-24T22:27:42.285 INFO:teuthology.orchestra.run.smithi080.stderr:pool 'rbd' created 2024-04-24T22:27:42.296 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-24T22:27:43.288 INFO:teuthology.orchestra.run.smithi080.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-24T22:27:43.300 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-04-24T22:27:43.301 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-04-24T22:27:43.301 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-04-24T22:27:43.301 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-04-24T22:27:43.301 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-24T22:27:43.670 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:43.682 INFO:teuthology.orchestra.run.smithi080.stdout:{"epoch":4,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6824","nonce":42261},{"type":"v1","addr":"172.21.15.80:6825","nonce":42261}]},"active_addr":"172.21.15.80:6825/42261","active_change":"2024-04-24T22:27:38.818201+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.80:0","nonce":404900425}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.80:0","nonce":2021933362}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.80:0","nonce":3979773315}]}]} 2024-04-24T22:27:43.686 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-04-24T22:27:43.687 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-04-24T22:27:43.687 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-24T22:27:44.000 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:44.001 INFO:teuthology.orchestra.run.smithi080.stdout:{"epoch":10,"fsid":"c6f15796-edf2-4aa3-acbd-00f4afb79b95","created":"2024-04-24T22:27:33.193401+0000","modified":"2024-04-24T22:27:43.277831+0000","last_up_change":"2024-04-24T22:27:41.275739+0000","last_in_change":"2024-04-24T22:27:34.169017+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-24T22:27:41.926799+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-24T22:27:42.860947+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"30b54047-4212-4616-bdcb-7ed629fbd802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":42679},{"type":"v1","addr":"172.21.15.80:6817","nonce":42679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":42679},{"type":"v1","addr":"172.21.15.80:6819","nonce":42679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":42679},{"type":"v1","addr":"172.21.15.80:6823","nonce":42679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":42679},{"type":"v1","addr":"172.21.15.80:6821","nonce":42679}]},"public_addr":"172.21.15.80:6817/42679","cluster_addr":"172.21.15.80:6819/42679","heartbeat_back_addr":"172.21.15.80:6823/42679","heartbeat_front_addr":"172.21.15.80:6821/42679","state":["exists","up"]},{"osd":1,"uuid":"3137d80f-296e-4fd9-a77f-97031fbe79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":42674},{"type":"v1","addr":"172.21.15.80:6809","nonce":42674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":42674},{"type":"v1","addr":"172.21.15.80:6811","nonce":42674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":42674},{"type":"v1","addr":"172.21.15.80:6815","nonce":42674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":42674},{"type":"v1","addr":"172.21.15.80:6813","nonce":42674}]},"public_addr":"172.21.15.80:6809/42674","cluster_addr":"172.21.15.80:6811/42674","heartbeat_back_addr":"172.21.15.80:6815/42674","heartbeat_front_addr":"172.21.15.80:6813/42674","state":["exists","up"]},{"osd":2,"uuid":"f5cd2fbf-7851-4596-bdee-30b92841e8e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":42673},{"type":"v1","addr":"172.21.15.80:6801","nonce":42673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":42673},{"type":"v1","addr":"172.21.15.80:6803","nonce":42673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":42673},{"type":"v1","addr":"172.21.15.80:6807","nonce":42673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":42673},{"type":"v1","addr":"172.21.15.80:6805","nonce":42673}]},"public_addr":"172.21.15.80:6801/42673","cluster_addr":"172.21.15.80:6803/42673","heartbeat_back_addr":"172.21.15.80:6807/42673","heartbeat_front_addr":"172.21.15.80:6805/42673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:27:39.347037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:27:39.373428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:27:39.360472+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T22:27:44.014 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-04-24T22:27:44.014 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-24T22:27:44.328 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:44.328 INFO:teuthology.orchestra.run.smithi080.stdout:{"epoch":11,"fsid":"c6f15796-edf2-4aa3-acbd-00f4afb79b95","created":"2024-04-24T22:27:33.193401+0000","modified":"2024-04-24T22:27:44.278556+0000","last_up_change":"2024-04-24T22:27:41.275739+0000","last_in_change":"2024-04-24T22:27:34.169017+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-24T22:27:41.926799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-24T22:27:42.860947+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"30b54047-4212-4616-bdcb-7ed629fbd802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":42679},{"type":"v1","addr":"172.21.15.80:6817","nonce":42679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":42679},{"type":"v1","addr":"172.21.15.80:6819","nonce":42679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":42679},{"type":"v1","addr":"172.21.15.80:6823","nonce":42679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":42679},{"type":"v1","addr":"172.21.15.80:6821","nonce":42679}]},"public_addr":"172.21.15.80:6817/42679","cluster_addr":"172.21.15.80:6819/42679","heartbeat_back_addr":"172.21.15.80:6823/42679","heartbeat_front_addr":"172.21.15.80:6821/42679","state":["exists","up"]},{"osd":1,"uuid":"3137d80f-296e-4fd9-a77f-97031fbe79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":42674},{"type":"v1","addr":"172.21.15.80:6809","nonce":42674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":42674},{"type":"v1","addr":"172.21.15.80:6811","nonce":42674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":42674},{"type":"v1","addr":"172.21.15.80:6815","nonce":42674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":42674},{"type":"v1","addr":"172.21.15.80:6813","nonce":42674}]},"public_addr":"172.21.15.80:6809/42674","cluster_addr":"172.21.15.80:6811/42674","heartbeat_back_addr":"172.21.15.80:6815/42674","heartbeat_front_addr":"172.21.15.80:6813/42674","state":["exists","up"]},{"osd":2,"uuid":"f5cd2fbf-7851-4596-bdee-30b92841e8e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":42673},{"type":"v1","addr":"172.21.15.80:6801","nonce":42673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":42673},{"type":"v1","addr":"172.21.15.80:6803","nonce":42673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":42673},{"type":"v1","addr":"172.21.15.80:6807","nonce":42673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":42673},{"type":"v1","addr":"172.21.15.80:6805","nonce":42673}]},"public_addr":"172.21.15.80:6801/42673","cluster_addr":"172.21.15.80:6803/42673","heartbeat_back_addr":"172.21.15.80:6807/42673","heartbeat_front_addr":"172.21.15.80:6805/42673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:27:39.347037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:27:39.373428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:27:39.360472+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T22:27:44.341 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-04-24T22:27:44.342 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-04-24T22:27:44.342 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-04-24T22:27:44.528 INFO:teuthology.orchestra.run.smithi080.stdout:34359738371 2024-04-24T22:27:44.529 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-04-24T22:27:44.536 INFO:teuthology.orchestra.run.smithi080.stdout:34359738371 2024-04-24T22:27:44.536 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-04-24T22:27:44.604 INFO:teuthology.orchestra.run.smithi080.stdout:34359738371 2024-04-24T22:27:44.605 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-04-24T22:27:44.931 INFO:teuthology.orchestra.run.smithi080.stdout:34359738370 2024-04-24T22:27:44.945 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738370 for osd.1 2024-04-24T22:27:44.993 INFO:teuthology.orchestra.run.smithi080.stdout:34359738370 2024-04-24T22:27:45.007 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738370 for osd.2 2024-04-24T22:27:45.041 INFO:teuthology.orchestra.run.smithi080.stdout:34359738370 2024-04-24T22:27:45.055 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738370 for osd.0 2024-04-24T22:27:45.946 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-04-24T22:27:46.008 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-04-24T22:27:46.056 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-04-24T22:27:46.302 INFO:teuthology.orchestra.run.smithi080.stdout:34359738371 2024-04-24T22:27:46.316 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.1 2024-04-24T22:27:46.316 DEBUG:teuthology.parallel:result is None 2024-04-24T22:27:46.349 INFO:teuthology.orchestra.run.smithi080.stdout:34359738371 2024-04-24T22:27:46.363 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.2 2024-04-24T22:27:46.363 DEBUG:teuthology.parallel:result is None 2024-04-24T22:27:46.391 INFO:teuthology.orchestra.run.smithi080.stdout:34359738371 2024-04-24T22:27:46.404 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.0 2024-04-24T22:27:46.404 DEBUG:teuthology.parallel:result is None 2024-04-24T22:27:46.405 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-04-24T22:27:46.405 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-24T22:27:46.753 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:46.755 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-04-24T22:27:46.766 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-24T22:27:45.280158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":892348,"kb_used_data":256,"kb_used_omap":4,"kb_used_meta":892027,"kb_avail":282223172,"statfs":{"total":289910292480,"available":288996528128,"internally_reserved":0,"allocated":262144,"data_stored":63744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":913435656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.457107"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-24T22:27:43.289967+0000","last_change":"2024-04-24T22:27:43.285088+0000","last_active":"2024-04-24T22:27:43.289967+0000","last_peered":"2024-04-24T22:27:43.289967+0000","last_clean":"2024-04-24T22:27:43.289967+0000","last_became_active":"2024-04-24T22:27:43.284828+0000","last_became_peered":"2024-04-24T22:27:43.284828+0000","last_unstale":"2024-04-24T22:27:43.289967+0000","last_undegraded":"2024-04-24T22:27:43.289967+0000","last_fullsized":"2024-04-24T22:27:43.289967+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:49:43.635606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-24T22:27:43.290058+0000","last_change":"2024-04-24T22:27:43.286377+0000","last_active":"2024-04-24T22:27:43.290058+0000","last_peered":"2024-04-24T22:27:43.290058+0000","last_clean":"2024-04-24T22:27:43.290058+0000","last_became_active":"2024-04-24T22:27:43.286126+0000","last_became_peered":"2024-04-24T22:27:43.286126+0000","last_unstale":"2024-04-24T22:27:43.290058+0000","last_undegraded":"2024-04-24T22:27:43.290058+0000","last_fullsized":"2024-04-24T22:27:43.290058+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:31:54.138109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-24T22:27:43.287460+0000","last_change":"2024-04-24T22:27:43.286455+0000","last_active":"2024-04-24T22:27:43.287460+0000","last_peered":"2024-04-24T22:27:43.287460+0000","last_clean":"2024-04-24T22:27:43.287460+0000","last_became_active":"2024-04-24T22:27:43.286197+0000","last_became_peered":"2024-04-24T22:27:43.286197+0000","last_unstale":"2024-04-24T22:27:43.287460+0000","last_undegraded":"2024-04-24T22:27:43.287460+0000","last_fullsized":"2024-04-24T22:27:43.287460+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:04:48.931409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-24T22:27:43.290032+0000","last_change":"2024-04-24T22:27:43.285898+0000","last_active":"2024-04-24T22:27:43.290032+0000","last_peered":"2024-04-24T22:27:43.290032+0000","last_clean":"2024-04-24T22:27:43.290032+0000","last_became_active":"2024-04-24T22:27:43.285646+0000","last_became_peered":"2024-04-24T22:27:43.285646+0000","last_unstale":"2024-04-24T22:27:43.290032+0000","last_undegraded":"2024-04-24T22:27:43.290032+0000","last_fullsized":"2024-04-24T22:27:43.290032+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:30.713659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-24T22:27:43.286697+0000","last_change":"2024-04-24T22:27:43.285244+0000","last_active":"2024-04-24T22:27:43.286697+0000","last_peered":"2024-04-24T22:27:43.286697+0000","last_clean":"2024-04-24T22:27:43.286697+0000","last_became_active":"2024-04-24T22:27:43.284970+0000","last_became_peered":"2024-04-24T22:27:43.284970+0000","last_unstale":"2024-04-24T22:27:43.286697+0000","last_undegraded":"2024-04-24T22:27:43.286697+0000","last_fullsized":"2024-04-24T22:27:43.286697+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:07:27.079460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"11'64","reported_seq":96,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-24T22:27:44.487530+0000","last_change":"2024-04-24T22:27:44.347665+0000","last_active":"2024-04-24T22:27:44.487530+0000","last_peered":"2024-04-24T22:27:44.487530+0000","last_clean":"2024-04-24T22:27:44.487530+0000","last_became_active":"2024-04-24T22:27:44.346752+0000","last_became_peered":"2024-04-24T22:27:44.346752+0000","last_unstale":"2024-04-24T22:27:44.487530+0000","last_undegraded":"2024-04-24T22:27:44.487530+0000","last_fullsized":"2024-04-24T22:27:44.487530+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:43.277831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:43.277831+0000","last_clean_scrub_stamp":"2024-04-24T22:27:43.277831+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:48:56.331694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-24T22:27:43.289973+0000","last_change":"2024-04-24T22:27:43.285195+0000","last_active":"2024-04-24T22:27:43.289973+0000","last_peered":"2024-04-24T22:27:43.289973+0000","last_clean":"2024-04-24T22:27:43.289973+0000","last_became_active":"2024-04-24T22:27:43.284939+0000","last_became_peered":"2024-04-24T22:27:43.284939+0000","last_unstale":"2024-04-24T22:27:43.289973+0000","last_undegraded":"2024-04-24T22:27:43.289973+0000","last_fullsized":"2024-04-24T22:27:43.289973+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:17.500091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-24T22:27:43.290081+0000","last_change":"2024-04-24T22:27:43.286584+0000","last_active":"2024-04-24T22:27:43.290081+0000","last_peered":"2024-04-24T22:27:43.290081+0000","last_clean":"2024-04-24T22:27:43.290081+0000","last_became_active":"2024-04-24T22:27:43.285821+0000","last_became_peered":"2024-04-24T22:27:43.285821+0000","last_unstale":"2024-04-24T22:27:43.290081+0000","last_undegraded":"2024-04-24T22:27:43.290081+0000","last_fullsized":"2024-04-24T22:27:43.290081+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:59.162446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-24T22:27:43.287441+0000","last_change":"2024-04-24T22:27:43.285734+0000","last_active":"2024-04-24T22:27:43.287441+0000","last_peered":"2024-04-24T22:27:43.287441+0000","last_clean":"2024-04-24T22:27:43.287441+0000","last_became_active":"2024-04-24T22:27:43.285451+0000","last_became_peered":"2024-04-24T22:27:43.285451+0000","last_unstale":"2024-04-24T22:27:43.287441+0000","last_undegraded":"2024-04-24T22:27:43.287441+0000","last_fullsized":"2024-04-24T22:27:43.287441+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:05:25.557870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":918560,"data_compressed":4980,"data_compressed_allocated":57344,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297448,"kb_used_data":84,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074392,"statfs":{"total":96636764160,"available":96332177408,"internally_reserved":0,"allocated":86016,"data_stored":21022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297448,"kb_used_data":84,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074392,"statfs":{"total":96636764160,"available":96332177408,"internally_reserved":0,"allocated":86016,"data_stored":21022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297452,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074388,"statfs":{"total":96636764160,"available":96332173312,"internally_reserved":0,"allocated":90112,"data_stored":21700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.216}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":459280,"data_compressed":2490,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":459280,"data_compressed":2490,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T22:27:46.767 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-24T22:27:47.073 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:47.075 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-04-24T22:27:47.086 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-04-24T22:27:46.823319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":892348,"kb_used_data":256,"kb_used_omap":4,"kb_used_meta":892027,"kb_avail":282223172,"statfs":{"total":289910292480,"available":288996528128,"internally_reserved":0,"allocated":262144,"data_stored":63744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":913435656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000268"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-24T22:27:43.289967+0000","last_change":"2024-04-24T22:27:43.285088+0000","last_active":"2024-04-24T22:27:43.289967+0000","last_peered":"2024-04-24T22:27:43.289967+0000","last_clean":"2024-04-24T22:27:43.289967+0000","last_became_active":"2024-04-24T22:27:43.284828+0000","last_became_peered":"2024-04-24T22:27:43.284828+0000","last_unstale":"2024-04-24T22:27:43.289967+0000","last_undegraded":"2024-04-24T22:27:43.289967+0000","last_fullsized":"2024-04-24T22:27:43.289967+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:49:43.635606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-24T22:27:43.290058+0000","last_change":"2024-04-24T22:27:43.286377+0000","last_active":"2024-04-24T22:27:43.290058+0000","last_peered":"2024-04-24T22:27:43.290058+0000","last_clean":"2024-04-24T22:27:43.290058+0000","last_became_active":"2024-04-24T22:27:43.286126+0000","last_became_peered":"2024-04-24T22:27:43.286126+0000","last_unstale":"2024-04-24T22:27:43.290058+0000","last_undegraded":"2024-04-24T22:27:43.290058+0000","last_fullsized":"2024-04-24T22:27:43.290058+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:31:54.138109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-24T22:27:43.287460+0000","last_change":"2024-04-24T22:27:43.286455+0000","last_active":"2024-04-24T22:27:43.287460+0000","last_peered":"2024-04-24T22:27:43.287460+0000","last_clean":"2024-04-24T22:27:43.287460+0000","last_became_active":"2024-04-24T22:27:43.286197+0000","last_became_peered":"2024-04-24T22:27:43.286197+0000","last_unstale":"2024-04-24T22:27:43.287460+0000","last_undegraded":"2024-04-24T22:27:43.287460+0000","last_fullsized":"2024-04-24T22:27:43.287460+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:04:48.931409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-24T22:27:43.290032+0000","last_change":"2024-04-24T22:27:43.285898+0000","last_active":"2024-04-24T22:27:43.290032+0000","last_peered":"2024-04-24T22:27:43.290032+0000","last_clean":"2024-04-24T22:27:43.290032+0000","last_became_active":"2024-04-24T22:27:43.285646+0000","last_became_peered":"2024-04-24T22:27:43.285646+0000","last_unstale":"2024-04-24T22:27:43.290032+0000","last_undegraded":"2024-04-24T22:27:43.290032+0000","last_fullsized":"2024-04-24T22:27:43.290032+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:30.713659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-24T22:27:43.286697+0000","last_change":"2024-04-24T22:27:43.285244+0000","last_active":"2024-04-24T22:27:43.286697+0000","last_peered":"2024-04-24T22:27:43.286697+0000","last_clean":"2024-04-24T22:27:43.286697+0000","last_became_active":"2024-04-24T22:27:43.284970+0000","last_became_peered":"2024-04-24T22:27:43.284970+0000","last_unstale":"2024-04-24T22:27:43.286697+0000","last_undegraded":"2024-04-24T22:27:43.286697+0000","last_fullsized":"2024-04-24T22:27:43.286697+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:07:27.079460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"11'64","reported_seq":96,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-24T22:27:44.487530+0000","last_change":"2024-04-24T22:27:44.347665+0000","last_active":"2024-04-24T22:27:44.487530+0000","last_peered":"2024-04-24T22:27:44.487530+0000","last_clean":"2024-04-24T22:27:44.487530+0000","last_became_active":"2024-04-24T22:27:44.346752+0000","last_became_peered":"2024-04-24T22:27:44.346752+0000","last_unstale":"2024-04-24T22:27:44.487530+0000","last_undegraded":"2024-04-24T22:27:44.487530+0000","last_fullsized":"2024-04-24T22:27:44.487530+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:43.277831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:43.277831+0000","last_clean_scrub_stamp":"2024-04-24T22:27:43.277831+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:48:56.331694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-24T22:27:43.289973+0000","last_change":"2024-04-24T22:27:43.285195+0000","last_active":"2024-04-24T22:27:43.289973+0000","last_peered":"2024-04-24T22:27:43.289973+0000","last_clean":"2024-04-24T22:27:43.289973+0000","last_became_active":"2024-04-24T22:27:43.284939+0000","last_became_peered":"2024-04-24T22:27:43.284939+0000","last_unstale":"2024-04-24T22:27:43.289973+0000","last_undegraded":"2024-04-24T22:27:43.289973+0000","last_fullsized":"2024-04-24T22:27:43.289973+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:17.500091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-24T22:27:43.290081+0000","last_change":"2024-04-24T22:27:43.286584+0000","last_active":"2024-04-24T22:27:43.290081+0000","last_peered":"2024-04-24T22:27:43.290081+0000","last_clean":"2024-04-24T22:27:43.290081+0000","last_became_active":"2024-04-24T22:27:43.285821+0000","last_became_peered":"2024-04-24T22:27:43.285821+0000","last_unstale":"2024-04-24T22:27:43.290081+0000","last_undegraded":"2024-04-24T22:27:43.290081+0000","last_fullsized":"2024-04-24T22:27:43.290081+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:59.162446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-24T22:27:43.287441+0000","last_change":"2024-04-24T22:27:43.285734+0000","last_active":"2024-04-24T22:27:43.287441+0000","last_peered":"2024-04-24T22:27:43.287441+0000","last_clean":"2024-04-24T22:27:43.287441+0000","last_became_active":"2024-04-24T22:27:43.285451+0000","last_became_peered":"2024-04-24T22:27:43.285451+0000","last_unstale":"2024-04-24T22:27:43.287441+0000","last_undegraded":"2024-04-24T22:27:43.287441+0000","last_fullsized":"2024-04-24T22:27:43.287441+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:05:25.557870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":918560,"data_compressed":4980,"data_compressed_allocated":57344,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297448,"kb_used_data":84,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074392,"statfs":{"total":96636764160,"available":96332177408,"internally_reserved":0,"allocated":86016,"data_stored":21022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297448,"kb_used_data":84,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074392,"statfs":{"total":96636764160,"available":96332177408,"internally_reserved":0,"allocated":86016,"data_stored":21022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297452,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074388,"statfs":{"total":96636764160,"available":96332173312,"internally_reserved":0,"allocated":90112,"data_stored":21700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.216}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":459280,"data_compressed":2490,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":459280,"data_compressed":2490,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T22:27:47.087 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-04-24T22:27:47.087 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-04-24T22:27:47.087 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-04-24T22:27:47.087 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-04-24T22:27:47.435 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:47.435 INFO:teuthology.orchestra.run.smithi080.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-24T22:27:47.449 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-04-24T22:27:47.449 INFO:teuthology.run_tasks:Running task ceph... 2024-04-24T22:27:47.460 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-04-24T22:27:47.460 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-24T22:27:47.460 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-24_21:25:53-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7672255', 'branch': 'wip-yuriw4-testing-20240423.151325-quincy', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': None, 'first_in_suite': False, 'job_id': '7672255', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2024-04-24_21:25:53-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuriw4-testing-20240423.151325-quincy'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuriw4-testing-20240423.151325-quincy', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9954, 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-yuriw4-testing-20240423.151325-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_330067dbce47655d1f762ef14d546d2fde16a6a9/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9', 'targets': {'smithi080.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFtIISksWdrVRHufao/k3yWyq3S/2VFVLmirMvq9bK3yYzjhKiiias2g1SEIIOlD4G0INN8vwRO7Xj2+HQW6lo0=', 'smithi134.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBHEp9XcSD+LRpP+1zpkogwKIriYSdDjcXeYzLZCqzOirEGLV4Wd+CCGg/fGq3EI+YNtR9rEpDMqf2Rhx3KfpQQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-24_21:25:53', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-24T22:27:47.461 DEBUG:teuthology.orchestra.run.smithi080:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-04-24T22:27:47.463 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-04-24T22:27:47.486 DEBUG:teuthology.orchestra.run.smithi080:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T22:27:47.522 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T22:27:47.557 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:27:47.557 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T22:27:47.618 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T22:27:47.618 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_1 2024-04-24T22:27:47.677 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T22:27:47.677 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T22:27:47.678 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 6h/6d Inode: 166491 Links: 1 2024-04-24T22:27:47.678 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T22:27:47.678 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T22:27:47.678 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-04-24 22:26:56.099210387 +0000 2024-04-24T22:27:47.678 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-04-24 22:23:13.266909320 +0000 2024-04-24T22:27:47.678 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-04-24 22:23:13.266909320 +0000 2024-04-24T22:27:47.678 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: - 2024-04-24T22:27:47.678 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T22:27:47.747 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-04-24T22:27:47.747 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-04-24T22:27:47.747 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000309731 s, 1.7 MB/s 2024-04-24T22:27:47.748 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T22:27:47.808 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_2 2024-04-24T22:27:47.868 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T22:27:47.868 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T22:27:47.868 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 6h/6d Inode: 165812 Links: 1 2024-04-24T22:27:47.868 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T22:27:47.868 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T22:27:47.868 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-04-24 22:26:56.100210364 +0000 2024-04-24T22:27:47.868 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-04-24 22:23:13.624901170 +0000 2024-04-24T22:27:47.868 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-04-24 22:23:13.624901170 +0000 2024-04-24T22:27:47.868 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: - 2024-04-24T22:27:47.869 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T22:27:47.935 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-04-24T22:27:47.935 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-04-24T22:27:47.935 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000235648 s, 2.2 MB/s 2024-04-24T22:27:47.936 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T22:27:47.995 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_3 2024-04-24T22:27:48.053 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T22:27:48.053 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T22:27:48.053 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 6h/6d Inode: 171126 Links: 1 2024-04-24T22:27:48.053 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T22:27:48.053 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T22:27:48.053 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-04-24 22:26:56.100210364 +0000 2024-04-24T22:27:48.053 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-04-24 22:23:13.976893157 +0000 2024-04-24T22:27:48.054 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-04-24 22:23:13.976893157 +0000 2024-04-24T22:27:48.054 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: - 2024-04-24T22:27:48.054 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T22:27:48.119 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-04-24T22:27:48.119 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-04-24T22:27:48.119 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000169986 s, 3.0 MB/s 2024-04-24T22:27:48.120 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T22:27:48.178 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_4 2024-04-24T22:27:48.236 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T22:27:48.237 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T22:27:48.237 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 6h/6d Inode: 166635 Links: 1 2024-04-24T22:27:48.237 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T22:27:48.237 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T22:27:48.237 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-04-24 22:26:56.100210364 +0000 2024-04-24T22:27:48.237 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-04-24 22:23:14.332885052 +0000 2024-04-24T22:27:48.237 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-04-24 22:23:14.332885052 +0000 2024-04-24T22:27:48.237 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: - 2024-04-24T22:27:48.237 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T22:27:48.302 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-04-24T22:27:48.302 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-04-24T22:27:48.302 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000201137 s, 2.5 MB/s 2024-04-24T22:27:48.303 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T22:27:48.361 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T22:27:48.361 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi134.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-24T22:27:48.361 INFO:tasks.ceph:Generating config... 2024-04-24T22:27:48.363 INFO:tasks.ceph:[client] debug rbd = 20 2024-04-24T22:27:48.363 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-04-24T22:27:48.363 INFO:tasks.ceph:[client] log to stderr = False 2024-04-24T22:27:48.363 INFO:tasks.ceph:[client] rbd default features = 125 2024-04-24T22:27:48.363 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2024-04-24T22:27:48.363 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:48.363 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:48.363 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:48.363 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:48.363 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:48.364 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:48.364 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:48.364 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:48.364 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-24T22:27:48.364 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-04-24T22:27:48.364 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-04-24T22:27:48.364 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-24T22:27:48.364 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-24T22:27:48.364 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-24T22:27:48.364 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-24T22:27:48.364 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-24T22:27:48.364 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-24T22:27:48.364 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2024-04-24T22:27:48.364 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-24T22:27:48.365 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-24T22:27:48.365 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-24T22:27:48.365 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-24T22:27:48.365 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-24T22:27:48.365 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-24T22:27:48.365 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-24T22:27:48.365 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-24T22:27:48.365 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-24T22:27:48.365 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-24T22:27:48.365 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-24T22:27:48.365 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-24T22:27:48.365 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-04-24T22:27:48.366 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-04-24T22:27:48.439 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster2.keyring 2024-04-24T22:27:48.441 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-04-24T22:27:48.522 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-04-24T22:27:48.590 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.134')] 2024-04-24T22:27:48.590 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.134', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-04-24T22:27:48.591 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:27:48.591 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T22:27:48.648 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.134 --print /home/ubuntu/cephtest/cluster2.monmap 2024-04-24T22:27:48.727 INFO:teuthology.orchestra.run.smithi134.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-04-24T22:27:48.728 INFO:teuthology.orchestra.run.smithi134.stdout:monmaptool: generated fsid c413118c-6cbc-48c8-982f-3c229c51647f 2024-04-24T22:27:48.728 INFO:teuthology.orchestra.run.smithi134.stdout:setting min_mon_release = octopus 2024-04-24T22:27:48.728 INFO:teuthology.orchestra.run.smithi134.stdout:epoch 0 2024-04-24T22:27:48.728 INFO:teuthology.orchestra.run.smithi134.stdout:fsid c413118c-6cbc-48c8-982f-3c229c51647f 2024-04-24T22:27:48.728 INFO:teuthology.orchestra.run.smithi134.stdout:last_changed 2024-04-24T22:27:48.727203+0000 2024-04-24T22:27:48.728 INFO:teuthology.orchestra.run.smithi134.stdout:created 2024-04-24T22:27:48.727203+0000 2024-04-24T22:27:48.728 INFO:teuthology.orchestra.run.smithi134.stdout:min_mon_release 15 (octopus) 2024-04-24T22:27:48.728 INFO:teuthology.orchestra.run.smithi134.stdout:election_strategy: 1 2024-04-24T22:27:48.728 INFO:teuthology.orchestra.run.smithi134.stdout:0: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.a 2024-04-24T22:27:48.728 INFO:teuthology.orchestra.run.smithi134.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-04-24T22:27:48.729 DEBUG:teuthology.orchestra.run.smithi134:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T22:27:48.788 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID c413118c-6cbc-48c8-982f-3c229c51647f... 2024-04-24T22:27:48.789 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-04-24T22:27:48.792 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-04-24T22:27:48.844 INFO:teuthology.orchestra.run.smithi080.stdout:[global] 2024-04-24T22:27:48.845 INFO:teuthology.orchestra.run.smithi080.stdout: chdir = "" 2024-04-24T22:27:48.845 INFO:teuthology.orchestra.run.smithi080.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T22:27:48.845 INFO:teuthology.orchestra.run.smithi080.stdout: auth supported = cephx 2024-04-24T22:27:48.845 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.845 INFO:teuthology.orchestra.run.smithi080.stdout: filestore xattr use omap = true 2024-04-24T22:27:48.845 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.845 INFO:teuthology.orchestra.run.smithi080.stdout: mon clock drift allowed = 1.000 2024-04-24T22:27:48.845 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.845 INFO:teuthology.orchestra.run.smithi080.stdout: osd crush chooseleaf type = 0 2024-04-24T22:27:48.845 INFO:teuthology.orchestra.run.smithi080.stdout: auth debug = true 2024-04-24T22:27:48.845 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.845 INFO:teuthology.orchestra.run.smithi080.stdout: ms die on old message = true 2024-04-24T22:27:48.845 INFO:teuthology.orchestra.run.smithi080.stdout: ms die on bug = true 2024-04-24T22:27:48.846 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.846 INFO:teuthology.orchestra.run.smithi080.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T22:27:48.846 INFO:teuthology.orchestra.run.smithi080.stdout: mon pg warn max object skew = 0 2024-04-24T22:27:48.846 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.846 INFO:teuthology.orchestra.run.smithi080.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T22:27:48.846 INFO:teuthology.orchestra.run.smithi080.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T22:27:48.846 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.846 INFO:teuthology.orchestra.run.smithi080.stdout: osd pool default size = 2 2024-04-24T22:27:48.846 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.846 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd allow primary affinity = true 2024-04-24T22:27:48.846 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd allow pg remap = true 2024-04-24T22:27:48.846 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on legacy crush tunables = false 2024-04-24T22:27:48.846 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on crush straw calc version zero = false 2024-04-24T22:27:48.846 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on no sortbitwise = false 2024-04-24T22:27:48.846 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on osd down out interval zero = false 2024-04-24T22:27:48.847 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on too few osds = false 2024-04-24T22:27:48.847 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T22:27:48.847 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T22:27:48.847 INFO:teuthology.orchestra.run.smithi080.stdout: mon_allow_pool_size_one = true 2024-04-24T22:27:48.847 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.847 INFO:teuthology.orchestra.run.smithi080.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T22:27:48.847 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.847 INFO:teuthology.orchestra.run.smithi080.stdout: osd default data pool replay window = 5 2024-04-24T22:27:48.847 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.847 INFO:teuthology.orchestra.run.smithi080.stdout: mon allow pool delete = true 2024-04-24T22:27:48.847 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.847 INFO:teuthology.orchestra.run.smithi080.stdout: mon cluster log file level = debug 2024-04-24T22:27:48.847 INFO:teuthology.orchestra.run.smithi080.stdout: debug asserts on shutdown = true 2024-04-24T22:27:48.847 INFO:teuthology.orchestra.run.smithi080.stdout: mon health detail to clog = false 2024-04-24T22:27:48.847 INFO:teuthology.orchestra.run.smithi080.stdout: mon host = 172.21.15.134 2024-04-24T22:27:48.847 INFO:teuthology.orchestra.run.smithi080.stdout: mon client directed command retry = 5 2024-04-24T22:27:48.848 INFO:teuthology.orchestra.run.smithi080.stdout: mon warn on pool no app = False 2024-04-24T22:27:48.848 INFO:teuthology.orchestra.run.smithi080.stdout: ms inject socket failures = 5000 2024-04-24T22:27:48.848 INFO:teuthology.orchestra.run.smithi080.stdout: fsid = c413118c-6cbc-48c8-982f-3c229c51647f 2024-04-24T22:27:48.848 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.848 INFO:teuthology.orchestra.run.smithi080.stdout:[osd] 2024-04-24T22:27:48.848 INFO:teuthology.orchestra.run.smithi080.stdout: osd journal size = 100 2024-04-24T22:27:48.848 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.848 INFO:teuthology.orchestra.run.smithi080.stdout: osd scrub load threshold = 5.0 2024-04-24T22:27:48.848 INFO:teuthology.orchestra.run.smithi080.stdout: osd scrub max interval = 600 2024-04-24T22:27:48.848 INFO:teuthology.orchestra.run.smithi080.stdout: osd mclock profile = high_recovery_ops 2024-04-24T22:27:48.848 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.848 INFO:teuthology.orchestra.run.smithi080.stdout: osd recover clone overlap = true 2024-04-24T22:27:48.848 INFO:teuthology.orchestra.run.smithi080.stdout: osd recovery max chunk = 1048576 2024-04-24T22:27:48.848 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.848 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug shutdown = true 2024-04-24T22:27:48.848 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug op order = true 2024-04-24T22:27:48.849 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug verify stray on activate = true 2024-04-24T22:27:48.849 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.849 INFO:teuthology.orchestra.run.smithi080.stdout: osd open classes on start = true 2024-04-24T22:27:48.849 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug pg log writeout = true 2024-04-24T22:27:48.849 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.849 INFO:teuthology.orchestra.run.smithi080.stdout: osd deep scrub update digest min age = 30 2024-04-24T22:27:48.849 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.849 INFO:teuthology.orchestra.run.smithi080.stdout: osd map max advance = 10 2024-04-24T22:27:48.849 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.849 INFO:teuthology.orchestra.run.smithi080.stdout: journal zero on create = true 2024-04-24T22:27:48.849 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.849 INFO:teuthology.orchestra.run.smithi080.stdout: filestore ondisk finisher threads = 3 2024-04-24T22:27:48.849 INFO:teuthology.orchestra.run.smithi080.stdout: filestore apply finisher threads = 3 2024-04-24T22:27:48.849 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.849 INFO:teuthology.orchestra.run.smithi080.stdout: bdev debug aio = true 2024-04-24T22:27:48.850 INFO:teuthology.orchestra.run.smithi080.stdout: osd debug misdirected ops = true 2024-04-24T22:27:48.850 INFO:teuthology.orchestra.run.smithi080.stdout: bluestore block size = 96636764160 2024-04-24T22:27:48.850 INFO:teuthology.orchestra.run.smithi080.stdout: bluestore compression algorithm = lz4 2024-04-24T22:27:48.850 INFO:teuthology.orchestra.run.smithi080.stdout: bluestore compression mode = aggressive 2024-04-24T22:27:48.850 INFO:teuthology.orchestra.run.smithi080.stdout: bluestore fsck on mount = True 2024-04-24T22:27:48.850 INFO:teuthology.orchestra.run.smithi080.stdout: debug bluefs = 1/20 2024-04-24T22:27:48.850 INFO:teuthology.orchestra.run.smithi080.stdout: debug bluestore = 1/20 2024-04-24T22:27:48.850 INFO:teuthology.orchestra.run.smithi080.stdout: debug ms = 1 2024-04-24T22:27:48.850 INFO:teuthology.orchestra.run.smithi080.stdout: debug osd = 20 2024-04-24T22:27:48.850 INFO:teuthology.orchestra.run.smithi080.stdout: debug rocksdb = 4/10 2024-04-24T22:27:48.850 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T22:27:48.850 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd full ratio = 0.9 2024-04-24T22:27:48.850 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd nearfull ratio = 0.8 2024-04-24T22:27:48.850 INFO:teuthology.orchestra.run.smithi080.stdout: osd failsafe full ratio = 0.95 2024-04-24T22:27:48.850 INFO:teuthology.orchestra.run.smithi080.stdout: osd objectstore = bluestore 2024-04-24T22:27:48.850 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.851 INFO:teuthology.orchestra.run.smithi080.stdout:[mgr] 2024-04-24T22:27:48.851 INFO:teuthology.orchestra.run.smithi080.stdout: debug ms = 1 2024-04-24T22:27:48.851 INFO:teuthology.orchestra.run.smithi080.stdout: debug mgr = 20 2024-04-24T22:27:48.851 INFO:teuthology.orchestra.run.smithi080.stdout: debug mon = 20 2024-04-24T22:27:48.851 INFO:teuthology.orchestra.run.smithi080.stdout: debug auth = 20 2024-04-24T22:27:48.851 INFO:teuthology.orchestra.run.smithi080.stdout: mon reweight min pgs per osd = 4 2024-04-24T22:27:48.851 INFO:teuthology.orchestra.run.smithi080.stdout: mon reweight min bytes per osd = 10 2024-04-24T22:27:48.851 INFO:teuthology.orchestra.run.smithi080.stdout: mgr/telemetry/nag = false 2024-04-24T22:27:48.851 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.851 INFO:teuthology.orchestra.run.smithi080.stdout:[mon] 2024-04-24T22:27:48.851 INFO:teuthology.orchestra.run.smithi080.stdout: debug ms = 1 2024-04-24T22:27:48.851 INFO:teuthology.orchestra.run.smithi080.stdout: debug mon = 20 2024-04-24T22:27:48.851 INFO:teuthology.orchestra.run.smithi080.stdout: debug paxos = 20 2024-04-24T22:27:48.851 INFO:teuthology.orchestra.run.smithi080.stdout: debug auth = 20 2024-04-24T22:27:48.852 INFO:teuthology.orchestra.run.smithi080.stdout: mon data avail warn = 5 2024-04-24T22:27:48.852 INFO:teuthology.orchestra.run.smithi080.stdout: mon mgr mkfs grace = 240 2024-04-24T22:27:48.852 INFO:teuthology.orchestra.run.smithi080.stdout: mon reweight min pgs per osd = 4 2024-04-24T22:27:48.852 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd reporter subtree level = osd 2024-04-24T22:27:48.852 INFO:teuthology.orchestra.run.smithi080.stdout: mon osd prime pg temp = true 2024-04-24T22:27:48.852 INFO:teuthology.orchestra.run.smithi080.stdout: mon reweight min bytes per osd = 10 2024-04-24T22:27:48.852 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.852 INFO:teuthology.orchestra.run.smithi080.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T22:27:48.852 INFO:teuthology.orchestra.run.smithi080.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T22:27:48.852 INFO:teuthology.orchestra.run.smithi080.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T22:27:48.852 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.852 INFO:teuthology.orchestra.run.smithi080.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T22:27:48.852 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T22:27:48.852 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T22:27:48.852 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.853 INFO:teuthology.orchestra.run.smithi080.stdout: # 1m isn't quite enough 2024-04-24T22:27:48.853 INFO:teuthology.orchestra.run.smithi080.stdout: mon_down_mkfs_grace = 2m 2024-04-24T22:27:48.853 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.853 INFO:teuthology.orchestra.run.smithi080.stdout: mon_warn_on_filestore_osds = false 2024-04-24T22:27:48.853 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T22:27:48.853 INFO:teuthology.orchestra.run.smithi080.stdout:[client] 2024-04-24T22:27:48.853 INFO:teuthology.orchestra.run.smithi080.stdout: rgw cache enabled = true 2024-04-24T22:27:48.853 INFO:teuthology.orchestra.run.smithi080.stdout: rgw enable ops log = true 2024-04-24T22:27:48.853 INFO:teuthology.orchestra.run.smithi080.stdout: rgw enable usage log = true 2024-04-24T22:27:48.853 INFO:teuthology.orchestra.run.smithi080.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T22:27:48.853 INFO:teuthology.orchestra.run.smithi080.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T22:27:48.853 INFO:teuthology.orchestra.run.smithi080.stdout: debug rbd = 20 2024-04-24T22:27:48.853 INFO:teuthology.orchestra.run.smithi080.stdout: debug rbd_mirror = 15 2024-04-24T22:27:48.853 INFO:teuthology.orchestra.run.smithi080.stdout: log to stderr = False 2024-04-24T22:27:48.853 INFO:teuthology.orchestra.run.smithi080.stdout: rbd default features = 125 2024-04-24T22:27:48.853 INFO:teuthology.orchestra.run.smithi080.stdout: rbd mirror image policy type = none 2024-04-24T22:27:48.854 INFO:teuthology.orchestra.run.smithi080.stdout:[mon.a] 2024-04-24T22:27:48.854 INFO:teuthology.orchestra.run.smithi080.stdout:[client.mirror.0] 2024-04-24T22:27:48.854 INFO:teuthology.orchestra.run.smithi080.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:48.854 INFO:teuthology.orchestra.run.smithi080.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:48.854 INFO:teuthology.orchestra.run.smithi080.stdout:[client.mirror.1] 2024-04-24T22:27:48.854 INFO:teuthology.orchestra.run.smithi080.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:48.854 INFO:teuthology.orchestra.run.smithi080.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:48.854 INFO:teuthology.orchestra.run.smithi080.stdout:[client.mirror.2] 2024-04-24T22:27:48.854 INFO:teuthology.orchestra.run.smithi080.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:48.854 INFO:teuthology.orchestra.run.smithi080.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:48.854 INFO:teuthology.orchestra.run.smithi080.stdout:[client.mirror.3] 2024-04-24T22:27:48.854 INFO:teuthology.orchestra.run.smithi080.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:48.854 INFO:teuthology.orchestra.run.smithi080.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:48.872 INFO:teuthology.orchestra.run.smithi134.stdout:[global] 2024-04-24T22:27:48.872 INFO:teuthology.orchestra.run.smithi134.stdout: chdir = "" 2024-04-24T22:27:48.872 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T22:27:48.872 INFO:teuthology.orchestra.run.smithi134.stdout: auth supported = cephx 2024-04-24T22:27:48.872 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.872 INFO:teuthology.orchestra.run.smithi134.stdout: filestore xattr use omap = true 2024-04-24T22:27:48.872 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.872 INFO:teuthology.orchestra.run.smithi134.stdout: mon clock drift allowed = 1.000 2024-04-24T22:27:48.872 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.872 INFO:teuthology.orchestra.run.smithi134.stdout: osd crush chooseleaf type = 0 2024-04-24T22:27:48.873 INFO:teuthology.orchestra.run.smithi134.stdout: auth debug = true 2024-04-24T22:27:48.873 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.873 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on old message = true 2024-04-24T22:27:48.873 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on bug = true 2024-04-24T22:27:48.873 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.873 INFO:teuthology.orchestra.run.smithi134.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T22:27:48.873 INFO:teuthology.orchestra.run.smithi134.stdout: mon pg warn max object skew = 0 2024-04-24T22:27:48.873 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.873 INFO:teuthology.orchestra.run.smithi134.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T22:27:48.873 INFO:teuthology.orchestra.run.smithi134.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T22:27:48.873 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.873 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default size = 2 2024-04-24T22:27:48.873 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.873 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow primary affinity = true 2024-04-24T22:27:48.873 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow pg remap = true 2024-04-24T22:27:48.873 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on legacy crush tunables = false 2024-04-24T22:27:48.874 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on crush straw calc version zero = false 2024-04-24T22:27:48.874 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on no sortbitwise = false 2024-04-24T22:27:48.874 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on osd down out interval zero = false 2024-04-24T22:27:48.874 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on too few osds = false 2024-04-24T22:27:48.874 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T22:27:48.874 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T22:27:48.874 INFO:teuthology.orchestra.run.smithi134.stdout: mon_allow_pool_size_one = true 2024-04-24T22:27:48.874 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.874 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T22:27:48.874 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.874 INFO:teuthology.orchestra.run.smithi134.stdout: osd default data pool replay window = 5 2024-04-24T22:27:48.874 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.874 INFO:teuthology.orchestra.run.smithi134.stdout: mon allow pool delete = true 2024-04-24T22:27:48.874 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.874 INFO:teuthology.orchestra.run.smithi134.stdout: mon cluster log file level = debug 2024-04-24T22:27:48.875 INFO:teuthology.orchestra.run.smithi134.stdout: debug asserts on shutdown = true 2024-04-24T22:27:48.875 INFO:teuthology.orchestra.run.smithi134.stdout: mon health detail to clog = false 2024-04-24T22:27:48.875 INFO:teuthology.orchestra.run.smithi134.stdout: mon host = 172.21.15.134 2024-04-24T22:27:48.875 INFO:teuthology.orchestra.run.smithi134.stdout: mon client directed command retry = 5 2024-04-24T22:27:48.875 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on pool no app = False 2024-04-24T22:27:48.875 INFO:teuthology.orchestra.run.smithi134.stdout: ms inject socket failures = 5000 2024-04-24T22:27:48.875 INFO:teuthology.orchestra.run.smithi134.stdout: fsid = c413118c-6cbc-48c8-982f-3c229c51647f 2024-04-24T22:27:48.875 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.875 INFO:teuthology.orchestra.run.smithi134.stdout:[osd] 2024-04-24T22:27:48.875 INFO:teuthology.orchestra.run.smithi134.stdout: osd journal size = 100 2024-04-24T22:27:48.875 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.875 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub load threshold = 5.0 2024-04-24T22:27:48.875 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub max interval = 600 2024-04-24T22:27:48.875 INFO:teuthology.orchestra.run.smithi134.stdout: osd mclock profile = high_recovery_ops 2024-04-24T22:27:48.875 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.875 INFO:teuthology.orchestra.run.smithi134.stdout: osd recover clone overlap = true 2024-04-24T22:27:48.876 INFO:teuthology.orchestra.run.smithi134.stdout: osd recovery max chunk = 1048576 2024-04-24T22:27:48.876 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.876 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug shutdown = true 2024-04-24T22:27:48.876 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug op order = true 2024-04-24T22:27:48.876 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug verify stray on activate = true 2024-04-24T22:27:48.876 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.876 INFO:teuthology.orchestra.run.smithi134.stdout: osd open classes on start = true 2024-04-24T22:27:48.876 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug pg log writeout = true 2024-04-24T22:27:48.876 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.876 INFO:teuthology.orchestra.run.smithi134.stdout: osd deep scrub update digest min age = 30 2024-04-24T22:27:48.876 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.876 INFO:teuthology.orchestra.run.smithi134.stdout: osd map max advance = 10 2024-04-24T22:27:48.876 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.876 INFO:teuthology.orchestra.run.smithi134.stdout: journal zero on create = true 2024-04-24T22:27:48.876 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.876 INFO:teuthology.orchestra.run.smithi134.stdout: filestore ondisk finisher threads = 3 2024-04-24T22:27:48.876 INFO:teuthology.orchestra.run.smithi134.stdout: filestore apply finisher threads = 3 2024-04-24T22:27:48.877 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.877 INFO:teuthology.orchestra.run.smithi134.stdout: bdev debug aio = true 2024-04-24T22:27:48.877 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug misdirected ops = true 2024-04-24T22:27:48.877 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore block size = 96636764160 2024-04-24T22:27:48.877 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore compression algorithm = lz4 2024-04-24T22:27:48.877 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore compression mode = aggressive 2024-04-24T22:27:48.877 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore fsck on mount = True 2024-04-24T22:27:48.877 INFO:teuthology.orchestra.run.smithi134.stdout: debug bluefs = 1/20 2024-04-24T22:27:48.877 INFO:teuthology.orchestra.run.smithi134.stdout: debug bluestore = 1/20 2024-04-24T22:27:48.877 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-04-24T22:27:48.877 INFO:teuthology.orchestra.run.smithi134.stdout: debug osd = 20 2024-04-24T22:27:48.877 INFO:teuthology.orchestra.run.smithi134.stdout: debug rocksdb = 4/10 2024-04-24T22:27:48.877 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T22:27:48.877 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd full ratio = 0.9 2024-04-24T22:27:48.877 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd nearfull ratio = 0.8 2024-04-24T22:27:48.877 INFO:teuthology.orchestra.run.smithi134.stdout: osd failsafe full ratio = 0.95 2024-04-24T22:27:48.878 INFO:teuthology.orchestra.run.smithi134.stdout: osd objectstore = bluestore 2024-04-24T22:27:48.878 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.878 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr] 2024-04-24T22:27:48.878 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-04-24T22:27:48.878 INFO:teuthology.orchestra.run.smithi134.stdout: debug mgr = 20 2024-04-24T22:27:48.878 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-04-24T22:27:48.878 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-04-24T22:27:48.878 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-04-24T22:27:48.878 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-04-24T22:27:48.878 INFO:teuthology.orchestra.run.smithi134.stdout: mgr/telemetry/nag = false 2024-04-24T22:27:48.878 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.878 INFO:teuthology.orchestra.run.smithi134.stdout:[mon] 2024-04-24T22:27:48.878 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-04-24T22:27:48.878 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-04-24T22:27:48.878 INFO:teuthology.orchestra.run.smithi134.stdout: debug paxos = 20 2024-04-24T22:27:48.878 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-04-24T22:27:48.879 INFO:teuthology.orchestra.run.smithi134.stdout: mon data avail warn = 5 2024-04-24T22:27:48.879 INFO:teuthology.orchestra.run.smithi134.stdout: mon mgr mkfs grace = 240 2024-04-24T22:27:48.879 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-04-24T22:27:48.879 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd reporter subtree level = osd 2024-04-24T22:27:48.879 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd prime pg temp = true 2024-04-24T22:27:48.879 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-04-24T22:27:48.879 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.879 INFO:teuthology.orchestra.run.smithi134.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T22:27:48.879 INFO:teuthology.orchestra.run.smithi134.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T22:27:48.879 INFO:teuthology.orchestra.run.smithi134.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T22:27:48.879 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.879 INFO:teuthology.orchestra.run.smithi134.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T22:27:48.879 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T22:27:48.879 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T22:27:48.879 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.879 INFO:teuthology.orchestra.run.smithi134.stdout: # 1m isn't quite enough 2024-04-24T22:27:48.880 INFO:teuthology.orchestra.run.smithi134.stdout: mon_down_mkfs_grace = 2m 2024-04-24T22:27:48.880 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.880 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_filestore_osds = false 2024-04-24T22:27:48.880 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:27:48.880 INFO:teuthology.orchestra.run.smithi134.stdout:[client] 2024-04-24T22:27:48.880 INFO:teuthology.orchestra.run.smithi134.stdout: rgw cache enabled = true 2024-04-24T22:27:48.880 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable ops log = true 2024-04-24T22:27:48.880 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable usage log = true 2024-04-24T22:27:48.880 INFO:teuthology.orchestra.run.smithi134.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T22:27:48.880 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T22:27:48.880 INFO:teuthology.orchestra.run.smithi134.stdout: debug rbd = 20 2024-04-24T22:27:48.880 INFO:teuthology.orchestra.run.smithi134.stdout: debug rbd_mirror = 15 2024-04-24T22:27:48.880 INFO:teuthology.orchestra.run.smithi134.stdout: log to stderr = False 2024-04-24T22:27:48.880 INFO:teuthology.orchestra.run.smithi134.stdout: rbd default features = 125 2024-04-24T22:27:48.880 INFO:teuthology.orchestra.run.smithi134.stdout: rbd mirror image policy type = none 2024-04-24T22:27:48.880 INFO:teuthology.orchestra.run.smithi134.stdout:[mon.a] 2024-04-24T22:27:48.881 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.0] 2024-04-24T22:27:48.881 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:48.881 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:48.881 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.1] 2024-04-24T22:27:48.881 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:48.881 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:48.881 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.2] 2024-04-24T22:27:48.881 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:48.881 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:48.881 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.3] 2024-04-24T22:27:48.881 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T22:27:48.881 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T22:27:48.884 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-04-24T22:27:48.885 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-04-24T22:27:48.938 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-24T22:27:48.939 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:27:48.939 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-04-24T22:27:48.956 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:27:48.956 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-04-24T22:27:49.013 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi080.front.sepia.ceph.com 2024-04-24T22:27:49.013 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:27:49.013 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/ceph/cluster2.keyring 2024-04-24T22:27:49.013 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-04-24T22:27:49.057 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:27:49.057 DEBUG:teuthology.orchestra.run.smithi080:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-04-24T22:27:49.118 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi134.front.sepia.ceph.com 2024-04-24T22:27:49.118 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:27:49.118 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/cluster2.keyring 2024-04-24T22:27:49.119 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-04-24T22:27:49.154 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:27:49.154 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-04-24T22:27:49.214 INFO:tasks.ceph:Setting up mon nodes... 2024-04-24T22:27:49.214 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-24T22:27:49.215 DEBUG:teuthology.orchestra.run.smithi080:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-04-24T22:27:49.271 INFO:teuthology.orchestra.run.smithi080.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-04-24T22:27:49.273 INFO:tasks.ceph:Setting up mds nodes... 2024-04-24T22:27:49.274 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-24T22:27:49.274 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-04-24T22:27:49.317 INFO:teuthology.orchestra.run.smithi080.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-04-24T22:27:49.330 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-04-24T22:27:49.369 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-04-24T22:27:49.381 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-04-24T22:27:49.461 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-04-24T22:27:49.473 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-04-24T22:27:49.554 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-04-24T22:27:49.566 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-04-24T22:27:49.646 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-04-24T22:27:49.658 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-04-24T22:27:49.738 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-04-24T22:27:49.750 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-04-24T22:27:49.830 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-04-24T22:27:49.842 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-04-24T22:27:49.922 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-04-24T22:27:49.934 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-04-24T22:27:50.014 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-04-24T22:27:50.026 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-24T22:27:50.026 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi080.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi134.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-24T22:27:50.026 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-04-24T22:27:50.093 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T22:27:50.093 INFO:tasks.ceph:role: cluster2.osd.0 2024-04-24T22:27:50.093 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi134.front.sepia.ceph.com 2024-04-24T22:27:50.093 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-24T22:27:50.167 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T22:27:50.167 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T22:27:50.167 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T22:27:50.167 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T22:27:50.167 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T22:27:50.168 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-04-24T22:27:50.168 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T22:27:50.168 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T22:27:50.168 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T22:27:50.168 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T22:27:50.598 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-04-24T22:27:50.599 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-04-24T22:27:50.599 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-04-24T22:27:50.771 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-04-24T22:27:50.803 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-04-24T22:27:50.870 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T22:27:50.870 INFO:tasks.ceph:role: cluster2.osd.1 2024-04-24T22:27:50.870 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi134.front.sepia.ceph.com 2024-04-24T22:27:50.870 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-24T22:27:50.944 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T22:27:50.944 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T22:27:50.944 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T22:27:50.944 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T22:27:50.945 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T22:27:50.945 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-04-24T22:27:50.945 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T22:27:50.945 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T22:27:50.945 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T22:27:50.945 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T22:27:51.360 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-04-24T22:27:51.361 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-04-24T22:27:51.362 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-04-24T22:27:51.472 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-04-24T22:27:51.503 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-04-24T22:27:51.570 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T22:27:51.570 INFO:tasks.ceph:role: cluster2.osd.2 2024-04-24T22:27:51.570 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi134.front.sepia.ceph.com 2024-04-24T22:27:51.571 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-24T22:27:51.644 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T22:27:51.644 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T22:27:51.644 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T22:27:51.644 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T22:27:51.644 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T22:27:51.644 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-04-24T22:27:51.644 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T22:27:51.644 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T22:27:51.644 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T22:27:51.644 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T22:27:52.093 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-04-24T22:27:52.095 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-04-24T22:27:52.095 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-04-24T22:27:52.249 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-04-24T22:27:52.282 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-04-24T22:27:52.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:52.381+0000 7f30ffa83540 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-04-24T22:27:52.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:52.384+0000 7f30ffa83540 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-04-24T22:27:52.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:52.384+0000 7f30ffa83540 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-04-24T22:27:52.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:52.384+0000 7f30ffa83540 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-04-24T22:27:52.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:52.384+0000 7f30ffa83540 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-04-24T22:27:52.389 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:52.387+0000 7f30ffa83540 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-04-24T22:27:52.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:52.912+0000 7f30ffa83540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T22:27:53.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:53.678+0000 7f30ffa83540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T22:27:54.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:54.440+0000 7f30ffa83540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T22:27:55.235 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-04-24T22:27:55.267 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-04-24T22:27:55.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:55.352+0000 7f1eb1f21540 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-04-24T22:27:55.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:55.352+0000 7f1eb1f21540 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-04-24T22:27:55.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:55.353+0000 7f1eb1f21540 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-04-24T22:27:55.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:55.353+0000 7f1eb1f21540 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-04-24T22:27:55.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:55.353+0000 7f1eb1f21540 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-04-24T22:27:55.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:55.353+0000 7f1eb1f21540 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-04-24T22:27:55.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:55.881+0000 7f1eb1f21540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T22:27:56.646 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:56.644+0000 7f1eb1f21540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T22:27:57.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:57.407+0000 7f1eb1f21540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T22:27:58.203 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-04-24T22:27:58.235 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-04-24T22:27:58.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:58.321+0000 7f8ed5871540 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-04-24T22:27:58.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:58.322+0000 7f8ed5871540 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-04-24T22:27:58.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:58.322+0000 7f8ed5871540 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-04-24T22:27:58.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:58.322+0000 7f8ed5871540 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-04-24T22:27:58.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:58.322+0000 7f8ed5871540 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-04-24T22:27:58.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:58.322+0000 7f8ed5871540 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-04-24T22:27:58.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:58.847+0000 7f8ed5871540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T22:27:59.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:27:59.608+0000 7f8ed5871540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T22:28:00.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-04-24T22:28:00.370+0000 7f8ed5871540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T22:28:01.160 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-04-24T22:28:01.188 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-24T22:28:01.189 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:28:01.189 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-04-24T22:28:01.222 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:28:01.222 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-04-24T22:28:01.257 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:28:01.257 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-04-24T22:28:01.324 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:28:01.324 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-04-24T22:28:01.391 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-04-24T22:28:01.391 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-04-24T22:28:01.412 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:28:01.412 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-04-24T22:28:01.449 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:28:01.449 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-04-24T22:28:01.506 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:28:01.506 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-04-24T22:28:01.563 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:28:01.563 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-04-24T22:28:01.623 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:28:01.623 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-04-24T22:28:01.680 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:28:01.680 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-04-24T22:28:01.737 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:28:01.737 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-04-24T22:28:01.795 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:28:01.795 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-04-24T22:28:01.852 INFO:tasks.ceph:Adding keys to all mons... 2024-04-24T22:28:01.853 DEBUG:teuthology.orchestra.run.smithi134:> sudo tee -a /etc/ceph/cluster2.keyring 2024-04-24T22:28:01.918 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr.x] 2024-04-24T22:28:01.918 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQBlhylmFlMkEBAAt4nfK45S6YNRbH3WOkGU4A== 2024-04-24T22:28:01.918 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.0] 2024-04-24T22:28:01.918 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQBohylm+p3SFhAAFnEU4Pj4IczLHfJaeNXnoQ== 2024-04-24T22:28:01.918 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.1] 2024-04-24T22:28:01.918 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQBrhylmQ4giFRAAzNKF4ztbzBIPX2H/ZSb5+g== 2024-04-24T22:28:01.918 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.2] 2024-04-24T22:28:01.918 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQBuhylm0VI9ExAAqpbZHr75vsvnF6FtHH9/YQ== 2024-04-24T22:28:01.918 INFO:teuthology.orchestra.run.smithi134.stdout:[client.0] 2024-04-24T22:28:01.918 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQBlhylmhlnmEhAAuDYtz9tJ0tcKNFxzxQa54Q== 2024-04-24T22:28:01.918 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror] 2024-04-24T22:28:01.919 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQBlhylmRkgCFhAAupK4RIdlaW92cCCcWIosCw== 2024-04-24T22:28:01.919 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.0] 2024-04-24T22:28:01.919 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQBlhylmSUJ9GxAAfJ7UV0Cei8qHu+HqaXB4hQ== 2024-04-24T22:28:01.919 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.1] 2024-04-24T22:28:01.919 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQBlhylmYqkDIRAAbBX2ynm4TdHiclq3q4fjOw== 2024-04-24T22:28:01.919 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.2] 2024-04-24T22:28:01.919 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQBlhylm01iCJhAANDBwT+SWS8m/kqkjrNrsug== 2024-04-24T22:28:01.919 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.3] 2024-04-24T22:28:01.919 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQBlhylmKFkBLBAA+gDxfu/nQnwaO1JchtBa3A== 2024-04-24T22:28:01.919 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.4] 2024-04-24T22:28:01.919 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQBlhylm92F4MRAAUfPvG5qm3OrCc3UgjP5Qcw== 2024-04-24T22:28:01.919 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.5] 2024-04-24T22:28:01.919 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQBlhylmjGD0NhAAVUXd4Ca4v5Saf69lG3DtBg== 2024-04-24T22:28:01.919 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.6] 2024-04-24T22:28:01.919 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQBmhylmbN3XABAAGgOcswwtXO7rY5kfxCflAg== 2024-04-24T22:28:01.920 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T22:28:02.011 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T22:28:02.060 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T22:28:02.110 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T22:28:02.159 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T22:28:02.212 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T22:28:02.264 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T22:28:02.317 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T22:28:02.369 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T22:28:02.421 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T22:28:02.474 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T22:28:02.526 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T22:28:02.582 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T22:28:02.634 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-24T22:28:02.634 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-04-24T22:28:02.661 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-04-24T22:28:02.755 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-04-24T22:28:02.782 DEBUG:teuthology.orchestra.run.smithi134:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-04-24T22:28:02.839 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-04-24T22:28:02.840 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-04-24T22:28:02.840 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-04-24T22:28:02.881 INFO:tasks.ceph.cluster2.mon.a:Started 2024-04-24T22:28:02.882 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-04-24T22:28:02.882 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-04-24T22:28:02.882 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-04-24T22:28:02.887 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-04-24T22:28:02.887 INFO:tasks.ceph:Setting crush tunables to default 2024-04-24T22:28:02.887 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster cluster2 osd crush tunables default 2024-04-24T22:28:03.086 INFO:teuthology.orchestra.run.smithi134.stderr:adjusted tunables profile to default 2024-04-24T22:28:03.097 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-04-24T22:28:03.097 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:28:03.097 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-04-24T22:28:03.104 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:03.102+0000 7f6f55c361c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T22:28:03.122 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:28:03.123 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-04-24T22:28:03.189 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:28:03.189 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-04-24T22:28:03.241 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:03.239+0000 7f6f55c361c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T22:28:03.257 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster cluster2 osd new 15dc0c96-fde9-4f51-a952-d5ed00c63156 0 2024-04-24T22:28:03.499 INFO:teuthology.orchestra.run.smithi134.stdout:0 2024-04-24T22:28:03.502 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:03.500+0000 7f6f55c361c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T22:28:03.511 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster cluster2 osd new f65bbd82-d35b-47ae-aa29-588569f95d9a 1 2024-04-24T22:28:03.713 INFO:teuthology.orchestra.run.smithi134.stdout:1 2024-04-24T22:28:03.725 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster cluster2 osd new f097213e-5822-4222-b4b8-a5d430a22da0 2 2024-04-24T22:28:03.916 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:03.914+0000 7f6f55c361c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T22:28:03.928 INFO:teuthology.orchestra.run.smithi134.stdout:2 2024-04-24T22:28:03.940 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-04-24T22:28:03.940 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-04-24T22:28:03.942 INFO:tasks.ceph.cluster2.osd.0:Started 2024-04-24T22:28:03.942 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-04-24T22:28:03.942 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-04-24T22:28:03.944 INFO:tasks.ceph.cluster2.osd.1:Started 2024-04-24T22:28:03.944 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-04-24T22:28:03.944 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-04-24T22:28:03.946 INFO:tasks.ceph.cluster2.osd.2:Started 2024-04-24T22:28:03.946 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-24T22:28:04.050 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:04.049+0000 7f6f55c361c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T22:28:04.144 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:28:04.144 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":5,"fsid":"c413118c-6cbc-48c8-982f-3c229c51647f","created":"2024-04-24T22:28:02.992996+0000","modified":"2024-04-24T22:28:03.926171+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T22:28:03.926171+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"15dc0c96-fde9-4f51-a952-d5ed00c63156","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f65bbd82-d35b-47ae-aa29-588569f95d9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f097213e-5822-4222-b4b8-a5d430a22da0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T22:28:04.156 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-04-24T22:28:04.157 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-24T22:28:04.378 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:04.376+0000 7f6f55c361c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T22:28:04.459 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-04-24T22:28:04.546 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T22:28:04.542+0000 7fb1ff3fd540 -1 Falling back to public interface 2024-04-24T22:28:04.548 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:28:04.547+0000 7f0f643f3540 -1 Falling back to public interface 2024-04-24T22:28:04.549 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T22:28:04.548+0000 7f059a838540 -1 Falling back to public interface 2024-04-24T22:28:04.642 INFO:teuthology.misc.health.smithi134.stdout: 2024-04-24T22:28:04.643 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":5,"fsid":"c413118c-6cbc-48c8-982f-3c229c51647f","created":"2024-04-24T22:28:02.992996+0000","modified":"2024-04-24T22:28:03.926171+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T22:28:03.926171+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"15dc0c96-fde9-4f51-a952-d5ed00c63156","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f65bbd82-d35b-47ae-aa29-588569f95d9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f097213e-5822-4222-b4b8-a5d430a22da0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T22:28:04.653 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-24T22:28:05.012 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:05.010+0000 7f6f55c361c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T22:28:05.081 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:05.079+0000 7f6f55c361c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T22:28:05.204 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:05.202+0000 7f6f55c361c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T22:28:05.271 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:05.270+0000 7f6f55c361c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T22:28:05.446 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:05.445+0000 7f6f55c361c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T22:28:05.520 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:05.518+0000 7f6f55c361c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T22:28:05.594 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:05.592+0000 7f6f55c361c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T22:28:05.668 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:05.666+0000 7f6f55c361c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T22:28:05.814 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:05.813+0000 7f6f55c361c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T22:28:05.890 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:05.889+0000 7f6f55c361c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T22:28:06.164 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:06.162+0000 7f6f55c361c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T22:28:06.231 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:06.230+0000 7f6f55c361c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T22:28:06.669 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:06.667+0000 7f6f55c361c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T22:28:06.814 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:06.812+0000 7f6f55c361c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T22:28:07.119 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:07.117+0000 7f6f55c361c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T22:28:07.454 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:07.452+0000 7f6f55c361c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T22:28:07.593 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:07.591+0000 7f6f55c361c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T22:28:07.734 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:07.732+0000 7f6f55c361c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T22:28:07.818 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:07.817+0000 7f6f55c361c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T22:28:07.966 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T22:28:07.965+0000 7fb1ff3fd540 -1 osd.0 0 log_to_monitors true 2024-04-24T22:28:08.008 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T22:28:08.007+0000 7f059a838540 -1 osd.1 0 log_to_monitors true 2024-04-24T22:28:08.025 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:28:08.024+0000 7f0f643f3540 -1 osd.2 0 log_to_monitors true 2024-04-24T22:28:09.802 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T22:28:09.801+0000 7f0592c40700 -1 osd.1 0 waiting for initial osdmap 2024-04-24T22:28:09.806 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T22:28:09.805+0000 7f058a234700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T22:28:09.849 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T22:28:09.848+0000 7fb1f6002700 -1 osd.0 0 waiting for initial osdmap 2024-04-24T22:28:09.853 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T22:28:09.852+0000 7fb1eedf9700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T22:28:10.706 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:28:10.705+0000 7f0f5aff8700 -1 osd.2 0 waiting for initial osdmap 2024-04-24T22:28:10.709 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:28:10.708+0000 7f0f53def700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T22:28:10.957 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-04-24T22:28:11.060 INFO:tasks.ceph.cluster2.mgr.x.smithi080.stderr:2024-04-24T22:28:11.058+0000 7f6f1a3ba700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-24T22:28:11.264 INFO:teuthology.misc.health.smithi134.stdout: 2024-04-24T22:28:11.264 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":8,"fsid":"c413118c-6cbc-48c8-982f-3c229c51647f","created":"2024-04-24T22:28:02.992996+0000","modified":"2024-04-24T22:28:10.020046+0000","last_up_change":"2024-04-24T22:28:10.020046+0000","last_in_change":"2024-04-24T22:28:03.926171+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"15dc0c96-fde9-4f51-a952-d5ed00c63156","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":43461},{"type":"v1","addr":"172.21.15.134:6801","nonce":43461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":43461},{"type":"v1","addr":"172.21.15.134:6803","nonce":43461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":43461},{"type":"v1","addr":"172.21.15.134:6807","nonce":43461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":43461},{"type":"v1","addr":"172.21.15.134:6805","nonce":43461}]},"public_addr":"172.21.15.134:6801/43461","cluster_addr":"172.21.15.134:6803/43461","heartbeat_back_addr":"172.21.15.134:6807/43461","heartbeat_front_addr":"172.21.15.134:6805/43461","state":["exists","up"]},{"osd":1,"uuid":"f65bbd82-d35b-47ae-aa29-588569f95d9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":43460},{"type":"v1","addr":"172.21.15.134:6812","nonce":43460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":43460},{"type":"v1","addr":"172.21.15.134:6816","nonce":43460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6821","nonce":43460},{"type":"v1","addr":"172.21.15.134:6823","nonce":43460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":43460},{"type":"v1","addr":"172.21.15.134:6819","nonce":43460}]},"public_addr":"172.21.15.134:6812/43460","cluster_addr":"172.21.15.134:6816/43460","heartbeat_back_addr":"172.21.15.134:6823/43460","heartbeat_front_addr":"172.21.15.134:6819/43460","state":["exists","up"]},{"osd":2,"uuid":"f097213e-5822-4222-b4b8-a5d430a22da0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T22:28:11.276 DEBUG:teuthology.misc:2 of 3 OSDs are up 2024-04-24T22:28:17.580 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-04-24T22:28:17.898 INFO:teuthology.misc.health.smithi134.stdout: 2024-04-24T22:28:17.898 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":12,"fsid":"c413118c-6cbc-48c8-982f-3c229c51647f","created":"2024-04-24T22:28:02.992996+0000","modified":"2024-04-24T22:28:14.859192+0000","last_up_change":"2024-04-24T22:28:11.706725+0000","last_in_change":"2024-04-24T22:28:03.926171+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T22:28:11.879120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"15dc0c96-fde9-4f51-a952-d5ed00c63156","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":43461},{"type":"v1","addr":"172.21.15.134:6801","nonce":43461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":43461},{"type":"v1","addr":"172.21.15.134:6803","nonce":43461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":43461},{"type":"v1","addr":"172.21.15.134:6807","nonce":43461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":43461},{"type":"v1","addr":"172.21.15.134:6805","nonce":43461}]},"public_addr":"172.21.15.134:6801/43461","cluster_addr":"172.21.15.134:6803/43461","heartbeat_back_addr":"172.21.15.134:6807/43461","heartbeat_front_addr":"172.21.15.134:6805/43461","state":["exists","up"]},{"osd":1,"uuid":"f65bbd82-d35b-47ae-aa29-588569f95d9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":43460},{"type":"v1","addr":"172.21.15.134:6812","nonce":43460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":43460},{"type":"v1","addr":"172.21.15.134:6816","nonce":43460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6821","nonce":43460},{"type":"v1","addr":"172.21.15.134:6823","nonce":43460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":43460},{"type":"v1","addr":"172.21.15.134:6819","nonce":43460}]},"public_addr":"172.21.15.134:6812/43460","cluster_addr":"172.21.15.134:6816/43460","heartbeat_back_addr":"172.21.15.134:6823/43460","heartbeat_front_addr":"172.21.15.134:6819/43460","state":["exists","up"]},{"osd":2,"uuid":"f097213e-5822-4222-b4b8-a5d430a22da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":43462},{"type":"v1","addr":"172.21.15.134:6809","nonce":43462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6811","nonce":43462},{"type":"v1","addr":"172.21.15.134:6813","nonce":43462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":43462},{"type":"v1","addr":"172.21.15.134:6822","nonce":43462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6815","nonce":43462},{"type":"v1","addr":"172.21.15.134:6817","nonce":43462}]},"public_addr":"172.21.15.134:6809/43462","cluster_addr":"172.21.15.134:6813/43462","heartbeat_back_addr":"172.21.15.134:6822/43462","heartbeat_front_addr":"172.21.15.134:6817/43462","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:28:08.917946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:28:08.976698+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:28:09.060104+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T22:28:17.910 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-24T22:28:17.911 INFO:tasks.ceph:Creating RBD pool 2024-04-24T22:28:17.911 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-04-24T22:28:18.881 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'rbd' created 2024-04-24T22:28:18.894 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-24T22:28:19.888 INFO:teuthology.orchestra.run.smithi134.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-24T22:28:19.901 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-04-24T22:28:19.901 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-04-24T22:28:19.901 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-04-24T22:28:19.901 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-04-24T22:28:19.902 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-24T22:28:20.273 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:28:20.286 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":5,"flags":0,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6826","nonce":47153},{"type":"v1","addr":"172.21.15.80:6827","nonce":47153}]},"active_addr":"172.21.15.80:6827/47153","active_change":"2024-04-24T22:28:09.832194+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.80:0","nonce":2214825526}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.80:0","nonce":111575819}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.80:0","nonce":223793377}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.80:0","nonce":2312105476}]}]} 2024-04-24T22:28:20.288 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-04-24T22:28:20.288 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-04-24T22:28:20.288 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-24T22:28:20.598 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:28:20.598 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":14,"fsid":"c413118c-6cbc-48c8-982f-3c229c51647f","created":"2024-04-24T22:28:02.992996+0000","modified":"2024-04-24T22:28:19.874200+0000","last_up_change":"2024-04-24T22:28:11.706725+0000","last_in_change":"2024-04-24T22:28:03.926171+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T22:28:11.879120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T22:28:18.236744+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"15dc0c96-fde9-4f51-a952-d5ed00c63156","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":43461},{"type":"v1","addr":"172.21.15.134:6801","nonce":43461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":43461},{"type":"v1","addr":"172.21.15.134:6803","nonce":43461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":43461},{"type":"v1","addr":"172.21.15.134:6807","nonce":43461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":43461},{"type":"v1","addr":"172.21.15.134:6805","nonce":43461}]},"public_addr":"172.21.15.134:6801/43461","cluster_addr":"172.21.15.134:6803/43461","heartbeat_back_addr":"172.21.15.134:6807/43461","heartbeat_front_addr":"172.21.15.134:6805/43461","state":["exists","up"]},{"osd":1,"uuid":"f65bbd82-d35b-47ae-aa29-588569f95d9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":43460},{"type":"v1","addr":"172.21.15.134:6812","nonce":43460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":43460},{"type":"v1","addr":"172.21.15.134:6816","nonce":43460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6821","nonce":43460},{"type":"v1","addr":"172.21.15.134:6823","nonce":43460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":43460},{"type":"v1","addr":"172.21.15.134:6819","nonce":43460}]},"public_addr":"172.21.15.134:6812/43460","cluster_addr":"172.21.15.134:6816/43460","heartbeat_back_addr":"172.21.15.134:6823/43460","heartbeat_front_addr":"172.21.15.134:6819/43460","state":["exists","up"]},{"osd":2,"uuid":"f097213e-5822-4222-b4b8-a5d430a22da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":43462},{"type":"v1","addr":"172.21.15.134:6809","nonce":43462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6811","nonce":43462},{"type":"v1","addr":"172.21.15.134:6813","nonce":43462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":43462},{"type":"v1","addr":"172.21.15.134:6822","nonce":43462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6815","nonce":43462},{"type":"v1","addr":"172.21.15.134:6817","nonce":43462}]},"public_addr":"172.21.15.134:6809/43462","cluster_addr":"172.21.15.134:6813/43462","heartbeat_back_addr":"172.21.15.134:6822/43462","heartbeat_front_addr":"172.21.15.134:6817/43462","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:28:08.917946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:28:08.976698+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:28:09.060104+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T22:28:20.612 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-04-24T22:28:20.612 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-24T22:28:20.925 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:28:20.925 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":15,"fsid":"c413118c-6cbc-48c8-982f-3c229c51647f","created":"2024-04-24T22:28:02.992996+0000","modified":"2024-04-24T22:28:20.876463+0000","last_up_change":"2024-04-24T22:28:11.706725+0000","last_in_change":"2024-04-24T22:28:03.926171+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T22:28:11.879120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T22:28:18.236744+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"15dc0c96-fde9-4f51-a952-d5ed00c63156","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":43461},{"type":"v1","addr":"172.21.15.134:6801","nonce":43461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":43461},{"type":"v1","addr":"172.21.15.134:6803","nonce":43461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":43461},{"type":"v1","addr":"172.21.15.134:6807","nonce":43461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":43461},{"type":"v1","addr":"172.21.15.134:6805","nonce":43461}]},"public_addr":"172.21.15.134:6801/43461","cluster_addr":"172.21.15.134:6803/43461","heartbeat_back_addr":"172.21.15.134:6807/43461","heartbeat_front_addr":"172.21.15.134:6805/43461","state":["exists","up"]},{"osd":1,"uuid":"f65bbd82-d35b-47ae-aa29-588569f95d9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":43460},{"type":"v1","addr":"172.21.15.134:6812","nonce":43460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":43460},{"type":"v1","addr":"172.21.15.134:6816","nonce":43460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6821","nonce":43460},{"type":"v1","addr":"172.21.15.134:6823","nonce":43460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":43460},{"type":"v1","addr":"172.21.15.134:6819","nonce":43460}]},"public_addr":"172.21.15.134:6812/43460","cluster_addr":"172.21.15.134:6816/43460","heartbeat_back_addr":"172.21.15.134:6823/43460","heartbeat_front_addr":"172.21.15.134:6819/43460","state":["exists","up"]},{"osd":2,"uuid":"f097213e-5822-4222-b4b8-a5d430a22da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":43462},{"type":"v1","addr":"172.21.15.134:6809","nonce":43462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6811","nonce":43462},{"type":"v1","addr":"172.21.15.134:6813","nonce":43462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":43462},{"type":"v1","addr":"172.21.15.134:6822","nonce":43462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6815","nonce":43462},{"type":"v1","addr":"172.21.15.134:6817","nonce":43462}]},"public_addr":"172.21.15.134:6809/43462","cluster_addr":"172.21.15.134:6813/43462","heartbeat_back_addr":"172.21.15.134:6822/43462","heartbeat_front_addr":"172.21.15.134:6817/43462","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:28:08.917946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:28:08.976698+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:28:09.060104+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T22:28:20.939 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-04-24T22:28:20.940 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-04-24T22:28:20.941 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-04-24T22:28:21.102 INFO:teuthology.orchestra.run.smithi134.stdout:34359738372 2024-04-24T22:28:21.103 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-04-24T22:28:21.104 INFO:teuthology.orchestra.run.smithi134.stdout:38654705668 2024-04-24T22:28:21.105 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-04-24T22:28:21.106 INFO:teuthology.orchestra.run.smithi134.stdout:34359738372 2024-04-24T22:28:21.106 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-04-24T22:28:21.473 INFO:teuthology.orchestra.run.smithi134.stdout:38654705667 2024-04-24T22:28:21.478 INFO:teuthology.orchestra.run.smithi134.stdout:34359738370 2024-04-24T22:28:21.483 INFO:teuthology.orchestra.run.smithi134.stdout:34359738370 2024-04-24T22:28:21.489 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705668 got 38654705667 for osd.2 2024-04-24T22:28:21.494 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738370 for osd.1 2024-04-24T22:28:21.499 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738370 for osd.0 2024-04-24T22:28:22.490 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-04-24T22:28:22.495 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-04-24T22:28:22.499 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-04-24T22:28:22.838 INFO:teuthology.orchestra.run.smithi134.stdout:38654705668 2024-04-24T22:28:22.852 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705668 got 38654705668 for osd.2 2024-04-24T22:28:22.852 DEBUG:teuthology.parallel:result is None 2024-04-24T22:28:22.874 INFO:teuthology.orchestra.run.smithi134.stdout:34359738372 2024-04-24T22:28:22.879 INFO:teuthology.orchestra.run.smithi134.stdout:34359738372 2024-04-24T22:28:22.889 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738372 for osd.1 2024-04-24T22:28:22.889 DEBUG:teuthology.parallel:result is None 2024-04-24T22:28:22.894 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738372 for osd.0 2024-04-24T22:28:22.894 DEBUG:teuthology.parallel:result is None 2024-04-24T22:28:22.894 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-04-24T22:28:22.894 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-24T22:28:23.242 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:28:23.244 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-04-24T22:28:23.256 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-04-24T22:28:21.838106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":892556,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":892027,"kb_avail":282222964,"statfs":{"total":289910292480,"available":288996315136,"internally_reserved":0,"allocated":483328,"data_stored":1282247,"data_compressed":11512,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":913435656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000644"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T22:28:19.886666+0000","last_change":"2024-04-24T22:28:19.886666+0000","last_active":"2024-04-24T22:28:19.886666+0000","last_peered":"2024-04-24T22:28:19.886666+0000","last_clean":"2024-04-24T22:28:19.886666+0000","last_became_active":"2024-04-24T22:28:19.886369+0000","last_became_peered":"2024-04-24T22:28:19.886369+0000","last_unstale":"2024-04-24T22:28:19.886666+0000","last_undegraded":"2024-04-24T22:28:19.886666+0000","last_fullsized":"2024-04-24T22:28:19.886666+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:19:47.445279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T22:28:19.884742+0000","last_change":"2024-04-24T22:28:19.884742+0000","last_active":"2024-04-24T22:28:19.884742+0000","last_peered":"2024-04-24T22:28:19.884742+0000","last_clean":"2024-04-24T22:28:19.884742+0000","last_became_active":"2024-04-24T22:28:19.884428+0000","last_became_peered":"2024-04-24T22:28:19.884428+0000","last_unstale":"2024-04-24T22:28:19.884742+0000","last_undegraded":"2024-04-24T22:28:19.884742+0000","last_fullsized":"2024-04-24T22:28:19.884742+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:12:29.302280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T22:28:19.886868+0000","last_change":"2024-04-24T22:28:19.886868+0000","last_active":"2024-04-24T22:28:19.886868+0000","last_peered":"2024-04-24T22:28:19.886868+0000","last_clean":"2024-04-24T22:28:19.886868+0000","last_became_active":"2024-04-24T22:28:19.886592+0000","last_became_peered":"2024-04-24T22:28:19.886592+0000","last_unstale":"2024-04-24T22:28:19.886868+0000","last_undegraded":"2024-04-24T22:28:19.886868+0000","last_fullsized":"2024-04-24T22:28:19.886868+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:50.108365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T22:28:19.886951+0000","last_change":"2024-04-24T22:28:19.885268+0000","last_active":"2024-04-24T22:28:19.886951+0000","last_peered":"2024-04-24T22:28:19.886951+0000","last_clean":"2024-04-24T22:28:19.886951+0000","last_became_active":"2024-04-24T22:28:19.884948+0000","last_became_peered":"2024-04-24T22:28:19.884948+0000","last_unstale":"2024-04-24T22:28:19.886951+0000","last_undegraded":"2024-04-24T22:28:19.886951+0000","last_fullsized":"2024-04-24T22:28:19.886951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:00.739359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":138,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T22:28:18.878963+0000","last_change":"2024-04-24T22:28:13.865092+0000","last_active":"2024-04-24T22:28:18.878963+0000","last_peered":"2024-04-24T22:28:18.878963+0000","last_clean":"2024-04-24T22:28:18.878963+0000","last_became_active":"2024-04-24T22:28:13.864081+0000","last_became_peered":"2024-04-24T22:28:13.864081+0000","last_unstale":"2024-04-24T22:28:18.878963+0000","last_undegraded":"2024-04-24T22:28:18.878963+0000","last_fullsized":"2024-04-24T22:28:18.878963+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:12.854421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:12.854421+0000","last_clean_scrub_stamp":"2024-04-24T22:28:12.854421+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:39:56.775553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T22:28:19.883883+0000","last_change":"2024-04-24T22:28:19.883883+0000","last_active":"2024-04-24T22:28:19.883883+0000","last_peered":"2024-04-24T22:28:19.883883+0000","last_clean":"2024-04-24T22:28:19.883883+0000","last_became_active":"2024-04-24T22:28:19.883590+0000","last_became_peered":"2024-04-24T22:28:19.883590+0000","last_unstale":"2024-04-24T22:28:19.883883+0000","last_undegraded":"2024-04-24T22:28:19.883883+0000","last_fullsized":"2024-04-24T22:28:19.883883+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:24.775003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T22:28:19.885278+0000","last_change":"2024-04-24T22:28:19.883851+0000","last_active":"2024-04-24T22:28:19.885278+0000","last_peered":"2024-04-24T22:28:19.885278+0000","last_clean":"2024-04-24T22:28:19.885278+0000","last_became_active":"2024-04-24T22:28:19.883549+0000","last_became_peered":"2024-04-24T22:28:19.883549+0000","last_unstale":"2024-04-24T22:28:19.885278+0000","last_undegraded":"2024-04-24T22:28:19.885278+0000","last_fullsized":"2024-04-24T22:28:19.885278+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:39:31.099684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T22:28:19.885719+0000","last_change":"2024-04-24T22:28:19.885719+0000","last_active":"2024-04-24T22:28:19.885719+0000","last_peered":"2024-04-24T22:28:19.885719+0000","last_clean":"2024-04-24T22:28:19.885719+0000","last_became_active":"2024-04-24T22:28:19.885439+0000","last_became_peered":"2024-04-24T22:28:19.885439+0000","last_unstale":"2024-04-24T22:28:19.885719+0000","last_undegraded":"2024-04-24T22:28:19.885719+0000","last_fullsized":"2024-04-24T22:28:19.885719+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:05:50.933126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T22:28:19.885044+0000","last_change":"2024-04-24T22:28:19.885044+0000","last_active":"2024-04-24T22:28:19.885044+0000","last_peered":"2024-04-24T22:28:19.885044+0000","last_clean":"2024-04-24T22:28:19.885044+0000","last_became_active":"2024-04-24T22:28:19.884405+0000","last_became_peered":"2024-04-24T22:28:19.884405+0000","last_unstale":"2024-04-24T22:28:19.885044+0000","last_undegraded":"2024-04-24T22:28:19.885044+0000","last_fullsized":"2024-04-24T22:28:19.885044+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:59:08.229402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11512,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":122880,"data_stored":33837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297536,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074304,"statfs":{"total":96636764160,"available":96332087296,"internally_reserved":0,"allocated":180224,"data_stored":624205,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297536,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074304,"statfs":{"total":96636764160,"available":96332087296,"internally_reserved":0,"allocated":180224,"data_stored":624205,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T22:28:23.256 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-24T22:28:23.560 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:28:23.563 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-04-24T22:28:23.573 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-04-24T22:28:21.838106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":892556,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":892027,"kb_avail":282222964,"statfs":{"total":289910292480,"available":288996315136,"internally_reserved":0,"allocated":483328,"data_stored":1282247,"data_compressed":11512,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":913435656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000644"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T22:28:19.886666+0000","last_change":"2024-04-24T22:28:19.886666+0000","last_active":"2024-04-24T22:28:19.886666+0000","last_peered":"2024-04-24T22:28:19.886666+0000","last_clean":"2024-04-24T22:28:19.886666+0000","last_became_active":"2024-04-24T22:28:19.886369+0000","last_became_peered":"2024-04-24T22:28:19.886369+0000","last_unstale":"2024-04-24T22:28:19.886666+0000","last_undegraded":"2024-04-24T22:28:19.886666+0000","last_fullsized":"2024-04-24T22:28:19.886666+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:19:47.445279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T22:28:19.884742+0000","last_change":"2024-04-24T22:28:19.884742+0000","last_active":"2024-04-24T22:28:19.884742+0000","last_peered":"2024-04-24T22:28:19.884742+0000","last_clean":"2024-04-24T22:28:19.884742+0000","last_became_active":"2024-04-24T22:28:19.884428+0000","last_became_peered":"2024-04-24T22:28:19.884428+0000","last_unstale":"2024-04-24T22:28:19.884742+0000","last_undegraded":"2024-04-24T22:28:19.884742+0000","last_fullsized":"2024-04-24T22:28:19.884742+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:12:29.302280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T22:28:19.886868+0000","last_change":"2024-04-24T22:28:19.886868+0000","last_active":"2024-04-24T22:28:19.886868+0000","last_peered":"2024-04-24T22:28:19.886868+0000","last_clean":"2024-04-24T22:28:19.886868+0000","last_became_active":"2024-04-24T22:28:19.886592+0000","last_became_peered":"2024-04-24T22:28:19.886592+0000","last_unstale":"2024-04-24T22:28:19.886868+0000","last_undegraded":"2024-04-24T22:28:19.886868+0000","last_fullsized":"2024-04-24T22:28:19.886868+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:50.108365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T22:28:19.886951+0000","last_change":"2024-04-24T22:28:19.885268+0000","last_active":"2024-04-24T22:28:19.886951+0000","last_peered":"2024-04-24T22:28:19.886951+0000","last_clean":"2024-04-24T22:28:19.886951+0000","last_became_active":"2024-04-24T22:28:19.884948+0000","last_became_peered":"2024-04-24T22:28:19.884948+0000","last_unstale":"2024-04-24T22:28:19.886951+0000","last_undegraded":"2024-04-24T22:28:19.886951+0000","last_fullsized":"2024-04-24T22:28:19.886951+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:00.739359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":138,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-24T22:28:18.878963+0000","last_change":"2024-04-24T22:28:13.865092+0000","last_active":"2024-04-24T22:28:18.878963+0000","last_peered":"2024-04-24T22:28:18.878963+0000","last_clean":"2024-04-24T22:28:18.878963+0000","last_became_active":"2024-04-24T22:28:13.864081+0000","last_became_peered":"2024-04-24T22:28:13.864081+0000","last_unstale":"2024-04-24T22:28:18.878963+0000","last_undegraded":"2024-04-24T22:28:18.878963+0000","last_fullsized":"2024-04-24T22:28:18.878963+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:12.854421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:12.854421+0000","last_clean_scrub_stamp":"2024-04-24T22:28:12.854421+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:39:56.775553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T22:28:19.883883+0000","last_change":"2024-04-24T22:28:19.883883+0000","last_active":"2024-04-24T22:28:19.883883+0000","last_peered":"2024-04-24T22:28:19.883883+0000","last_clean":"2024-04-24T22:28:19.883883+0000","last_became_active":"2024-04-24T22:28:19.883590+0000","last_became_peered":"2024-04-24T22:28:19.883590+0000","last_unstale":"2024-04-24T22:28:19.883883+0000","last_undegraded":"2024-04-24T22:28:19.883883+0000","last_fullsized":"2024-04-24T22:28:19.883883+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:24.775003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T22:28:19.885278+0000","last_change":"2024-04-24T22:28:19.883851+0000","last_active":"2024-04-24T22:28:19.885278+0000","last_peered":"2024-04-24T22:28:19.885278+0000","last_clean":"2024-04-24T22:28:19.885278+0000","last_became_active":"2024-04-24T22:28:19.883549+0000","last_became_peered":"2024-04-24T22:28:19.883549+0000","last_unstale":"2024-04-24T22:28:19.885278+0000","last_undegraded":"2024-04-24T22:28:19.885278+0000","last_fullsized":"2024-04-24T22:28:19.885278+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:39:31.099684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T22:28:19.885719+0000","last_change":"2024-04-24T22:28:19.885719+0000","last_active":"2024-04-24T22:28:19.885719+0000","last_peered":"2024-04-24T22:28:19.885719+0000","last_clean":"2024-04-24T22:28:19.885719+0000","last_became_active":"2024-04-24T22:28:19.885439+0000","last_became_peered":"2024-04-24T22:28:19.885439+0000","last_unstale":"2024-04-24T22:28:19.885719+0000","last_undegraded":"2024-04-24T22:28:19.885719+0000","last_fullsized":"2024-04-24T22:28:19.885719+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:05:50.933126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-24T22:28:19.885044+0000","last_change":"2024-04-24T22:28:19.885044+0000","last_active":"2024-04-24T22:28:19.885044+0000","last_peered":"2024-04-24T22:28:19.885044+0000","last_clean":"2024-04-24T22:28:19.885044+0000","last_became_active":"2024-04-24T22:28:19.884405+0000","last_became_peered":"2024-04-24T22:28:19.884405+0000","last_unstale":"2024-04-24T22:28:19.885044+0000","last_undegraded":"2024-04-24T22:28:19.885044+0000","last_fullsized":"2024-04-24T22:28:19.885044+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:59:08.229402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11512,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":122880,"data_stored":33837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297536,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074304,"statfs":{"total":96636764160,"available":96332087296,"internally_reserved":0,"allocated":180224,"data_stored":624205,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297536,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074304,"statfs":{"total":96636764160,"available":96332087296,"internally_reserved":0,"allocated":180224,"data_stored":624205,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T22:28:23.574 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-04-24T22:28:23.574 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-04-24T22:28:23.575 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-04-24T22:28:23.575 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-04-24T22:28:23.924 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T22:28:23.925 INFO:teuthology.orchestra.run.smithi134.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-24T22:28:23.939 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-04-24T22:28:23.939 INFO:teuthology.run_tasks:Running task exec... 2024-04-24T22:28:23.953 INFO:teuthology.task.exec:Executing custom commands... 2024-04-24T22:28:23.953 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi134.front.sepia.ceph.com 2024-04-24T22:28:23.954 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T22:28:24.350 INFO:teuthology.orchestra.run.smithi134.stderr:updated caps for client.mirror 2024-04-24T22:28:24.362 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T22:28:24.759 INFO:teuthology.orchestra.run.smithi134.stderr:updated caps for client.mirror.0 2024-04-24T22:28:24.771 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T22:28:25.172 INFO:teuthology.orchestra.run.smithi134.stderr:updated caps for client.mirror.1 2024-04-24T22:28:25.184 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T22:28:25.582 INFO:teuthology.orchestra.run.smithi134.stderr:updated caps for client.mirror.2 2024-04-24T22:28:25.594 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T22:28:25.994 INFO:teuthology.orchestra.run.smithi134.stderr:updated caps for client.mirror.3 2024-04-24T22:28:26.006 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T22:28:26.408 INFO:teuthology.orchestra.run.smithi134.stderr:updated caps for client.mirror 2024-04-24T22:28:26.419 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T22:28:26.822 INFO:teuthology.orchestra.run.smithi134.stderr:updated caps for client.mirror.0 2024-04-24T22:28:26.833 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T22:28:27.232 INFO:teuthology.orchestra.run.smithi134.stderr:updated caps for client.mirror.1 2024-04-24T22:28:27.243 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T22:28:27.643 INFO:teuthology.orchestra.run.smithi134.stderr:updated caps for client.mirror.2 2024-04-24T22:28:27.655 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T22:28:28.058 INFO:teuthology.orchestra.run.smithi134.stderr:updated caps for client.mirror.3 2024-04-24T22:28:28.070 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-24T22:28:28.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-24T22:28:28.088 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-24T22:28:28.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-24T22:28:28.090 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-24T22:28:28.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-24T22:28:28.112 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-24T22:28:28.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-24T22:28:28.114 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-24T22:28:28.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-24T22:28:28.123 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-24T22:28:28.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-24T22:28:28.125 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-24T22:28:28.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-04-24T22:28:28.134 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-04-24T22:28:28.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-04-24T22:28:28.136 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-24T22:28:28.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-24T22:28:28.147 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-24T22:28:28.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-24T22:28:28.150 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-24T22:28:28.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-04-24T22:28:28.207 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-04-24T22:28:28.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-04-24T22:28:28.209 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-24T22:28:28.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-04-24T22:28:28.218 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-04-24T22:28:28.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-04-24T22:28:28.221 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-24T22:28:28.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-04-24T22:28:28.231 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-04-24T22:28:28.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-04-24T22:28:28.234 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-04-24T22:28:28.256 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1713997708 2024-04-24T22:28:28.257 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-04-24T22:28:28.257 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-04-24T22:28:28.275 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1713997708 2024-04-24T22:28:28.275 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-04-24T22:28:28.275 INFO:teuthology.run_tasks:Running task workunit... 2024-04-24T22:28:28.351 INFO:tasks.workunit:Pulling workunits from ref 330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T22:28:28.352 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-24T22:28:28.352 DEBUG:teuthology.orchestra.run.smithi134:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-24T22:28:28.352 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2024-04-24T22:28:28.353 INFO:tasks.rbd_mirror_thrash:waiting for 79 secs before thrashing 2024-04-24T22:28:28.353 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2024-04-24T22:28:28.353 INFO:tasks.rbd_mirror_thrash:waiting for 94 secs before thrashing 2024-04-24T22:28:28.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:28:28.371 INFO:teuthology.orchestra.run.smithi134.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-04-24T22:28:28.371 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-24T22:28:28.429 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-24T22:28:28.429 DEBUG:teuthology.orchestra.run.smithi134:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-04-24T22:28:28.486 INFO:tasks.workunit:timeout=6h 2024-04-24T22:28:28.487 INFO:tasks.workunit:cleanup=True 2024-04-24T22:28:28.487 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T22:28:28.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-04-24T22:29:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Note: switching to '330067dbce47655d1f762ef14d546d2fde16a6a9'. 2024-04-24T22:29:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:29:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-24T22:29:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-24T22:29:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:state without impacting any branches by switching back to a branch. 2024-04-24T22:29:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:29:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-24T22:29:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-24T22:29:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:29:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: git switch -c 2024-04-24T22:29:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:29:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Or undo this operation with: 2024-04-24T22:29:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:29:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: git switch - 2024-04-24T22:29:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:29:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-24T22:29:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:29:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:HEAD is now at 330067dbce4 Merge PR #54172 into wip-yuriw-testing-20240423.151325-quincy 2024-04-24T22:29:14.761 DEBUG:teuthology.orchestra.run.smithi134:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-04-24T22:29:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-24T22:29:14.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-04-24T22:29:14.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-24T22:29:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-24T22:29:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-24T22:29:15.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-04-24T22:29:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-04-24T22:29:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-24T22:29:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-04-24T22:29:15.507 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-24T22:29:15.507 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-04-24T22:29:15.565 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-04-24T22:29:15.567 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-04-24T22:29:15.567 DEBUG:teuthology.orchestra.run.smithi134:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=330067dbce47655d1f762ef14d546d2fde16a6a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-04-24T22:29:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ IMAGE_COUNT=50 2024-04-24T22:29:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ export LOCKDEP=0 2024-04-24T22:29:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ LOCKDEP=0 2024-04-24T22:29:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-04-24T22:29:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-04-24T22:29:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ type xmlstarlet 2024-04-24T22:29:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ XMLSTARLET=xmlstarlet 2024-04-24T22:29:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ RBD_MIRROR_INSTANCES=4 2024-04-24T22:29:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CLUSTER1=cluster1 2024-04-24T22:29:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CLUSTER2=cluster2 2024-04-24T22:29:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ PEER_CLUSTER_SUFFIX= 2024-04-24T22:29:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ POOL=mirror 2024-04-24T22:29:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ PARENT_POOL=mirror_parent 2024-04-24T22:29:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ NS1=ns1 2024-04-24T22:29:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ NS2=ns2 2024-04-24T22:29:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ TEMPDIR= 2024-04-24T22:29:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ID=mirror 2024-04-24T22:29:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2024-04-24T22:29:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-04-24T22:29:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ MIRROR_POOL_MODE=image 2024-04-24T22:29:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ MIRROR_IMAGE_MODE=snapshot 2024-04-24T22:29:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-04-24T22:29:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS='--id mirror' 2024-04-24T22:29:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ LAST_MIRROR_INSTANCE=3 2024-04-24T22:29:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-04-24T22:29:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-04-24T22:29:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-04-24T22:29:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_BIN=. 2024-04-24T22:29:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_SRC=. 2024-04-24T22:29:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' -e CMakeCache.txt ']' 2024-04-24T22:29:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 -gt 0 ']' 2024-04-24T22:29:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ setup 2024-04-24T22:29:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local c 2024-04-24T22:29:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-04-24T22:29:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ setup_tempdir 2024-04-24T22:29:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:29:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mktemp -d 2024-04-24T22:29:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ TEMPDIR=/tmp/tmp.DCvHjwkiaM 2024-04-24T22:29:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -z 1 ']' 2024-04-24T22:29:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ setup_pools cluster1 cluster2 2024-04-24T22:29:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:29:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local remote_cluster=cluster2 2024-04-24T22:29:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mon_map_file 2024-04-24T22:29:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mon_addr 2024-04-24T22:29:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local admin_key_file 2024-04-24T22:29:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local uuid 2024-04-24T22:29:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T22:29:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-04-24T22:29:16.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:pool 'mirror' already exists 2024-04-24T22:29:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T22:29:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-04-24T22:29:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:pool 'mirror_parent' already exists 2024-04-24T22:29:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T22:29:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 pool init mirror 2024-04-24T22:29:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T22:29:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-04-24T22:29:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:29:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T22:29:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2024-04-24T22:29:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-04-24T22:29:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-04-24T22:29:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-04-24T22:29:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2024-04-24T22:29:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-04-24T22:29:23.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -z ']' 2024-04-24T22:29:23.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -z ']' 2024-04-24T22:29:23.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ peer_add cluster1 mirror cluster2 2024-04-24T22:29:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:29:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:29:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:29:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:29:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local client_cluster=cluster2 2024-04-24T22:29:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:29:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local remote_cluster=cluster2 2024-04-24T22:29:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local uuid_var_name 2024-04-24T22:29:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:29:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local error_code 2024-04-24T22:29:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local peer_uuid 2024-04-24T22:29:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 16 32 2024-04-24T22:29:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:29:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-04-24T22:29:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ peer_uuid=a56abb4b-e2ab-4eb0-9450-fa5d0a242dd6 2024-04-24T22:29:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=0 2024-04-24T22:29:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:29:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -eq 17 ']' 2024-04-24T22:29:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 0 -eq 0 2024-04-24T22:29:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:29:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:29:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-04-24T22:29:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:29:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:29:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror_parent 2024-04-24T22:29:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:29:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local client_cluster=cluster2 2024-04-24T22:29:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:29:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local remote_cluster=cluster2 2024-04-24T22:29:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local uuid_var_name 2024-04-24T22:29:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:29:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local error_code 2024-04-24T22:29:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local peer_uuid 2024-04-24T22:29:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 16 32 2024-04-24T22:29:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:29:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-04-24T22:29:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ peer_uuid=ebce59bf-8fae-4852-a495-05a331533a62 2024-04-24T22:29:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=0 2024-04-24T22:29:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:29:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -eq 17 ']' 2024-04-24T22:29:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 0 -eq 0 2024-04-24T22:29:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:29:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:29:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ setup_pools cluster2 cluster1 2024-04-24T22:29:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:29:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local remote_cluster=cluster1 2024-04-24T22:29:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mon_map_file 2024-04-24T22:29:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mon_addr 2024-04-24T22:29:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local admin_key_file 2024-04-24T22:29:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local uuid 2024-04-24T22:29:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T22:29:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-04-24T22:29:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:pool 'mirror' already exists 2024-04-24T22:29:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T22:29:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-04-24T22:29:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:pool 'mirror_parent' already exists 2024-04-24T22:29:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T22:29:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 pool init mirror 2024-04-24T22:29:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T22:29:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-04-24T22:29:28.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T22:29:28.224+0000 7ff7e2e8a840 -1 rbd::mirror::PoolReplayer: 0x55a7e6f87800 init: failed to initialize remote pool poller: (2) No such file or directory 2024-04-24T22:29:28.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T22:29:28.256+0000 7ff3dffef840 -1 rbd::mirror::PoolReplayer: 0x559a38b85800 init: failed to initialize remote pool poller: (2) No such file or directory 2024-04-24T22:29:28.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T22:29:28.260+0000 7ff750626840 -1 rbd::mirror::PoolReplayer: 0x555d6916bb00 init: failed to initialize remote pool poller: (2) No such file or directory 2024-04-24T22:29:28.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T22:29:28.269+0000 7f4a66c6c840 -1 rbd::mirror::PoolReplayer: 0x55dcfd617b00 init: failed to initialize remote pool poller: (2) No such file or directory 2024-04-24T22:29:28.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T22:29:28.290+0000 7ff3dffef840 -1 rbd::mirror::PoolReplayer: 0x559a3a362f00 init: failed to initialize remote pool poller: (2) No such file or directory 2024-04-24T22:29:28.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T22:29:28.306+0000 7ff750626840 -1 rbd::mirror::PoolReplayer: 0x555d6b598c00 init: failed to initialize remote pool poller: (2) No such file or directory 2024-04-24T22:29:28.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T22:29:28.471+0000 7ff7e2e8a840 -1 rbd::mirror::PoolReplayer: 0x55a7e8770f00 init: failed to initialize remote pool poller: (2) No such file or directory 2024-04-24T22:29:28.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T22:29:28.509+0000 7f4a66c6c840 -1 rbd::mirror::PoolReplayer: 0x55dcffa52c00 init: failed to initialize remote pool poller: (2) No such file or directory 2024-04-24T22:29:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:29:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T22:29:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2024-04-24T22:29:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-04-24T22:29:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-04-24T22:29:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-04-24T22:29:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2024-04-24T22:29:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-04-24T22:29:31.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -z ']' 2024-04-24T22:29:31.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -z ']' 2024-04-24T22:29:31.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ peer_add cluster2 mirror cluster1 2024-04-24T22:29:31.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:29:31.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:29:31.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:29:31.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:29:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local client_cluster=cluster1 2024-04-24T22:29:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:29:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local remote_cluster=cluster1 2024-04-24T22:29:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local uuid_var_name 2024-04-24T22:29:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:29:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local error_code 2024-04-24T22:29:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local peer_uuid 2024-04-24T22:29:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 16 32 2024-04-24T22:29:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:29:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-04-24T22:29:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ peer_uuid=dafc2e67-0125-463a-b064-3fe7e6b06aaa 2024-04-24T22:29:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=0 2024-04-24T22:29:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:29:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -eq 17 ']' 2024-04-24T22:29:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 0 -eq 0 2024-04-24T22:29:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:29:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:29:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-04-24T22:29:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:29:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:29:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror_parent 2024-04-24T22:29:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:29:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local client_cluster=cluster1 2024-04-24T22:29:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:29:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local remote_cluster=cluster1 2024-04-24T22:29:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local uuid_var_name 2024-04-24T22:29:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:29:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local error_code 2024-04-24T22:29:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local peer_uuid 2024-04-24T22:29:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 16 32 2024-04-24T22:29:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:29:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-04-24T22:29:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ peer_uuid=aaec76d3-21a3-44ad-a3ab-71c799b00b4a 2024-04-24T22:29:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=0 2024-04-24T22:29:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:29:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -eq 17 ']' 2024-04-24T22:29:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 0 -eq 0 2024-04-24T22:29:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:29:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:29:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:29:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ start_mirrors cluster1 2024-04-24T22:29:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:29:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T22:29:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ start_mirror cluster1:0 2024-04-24T22:29:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T22:29:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T22:29:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:29:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:0 2024-04-24T22:29:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T22:29:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T22:29:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T22:29:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T22:29:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T22:29:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T22:29:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T22:29:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:29:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T22:29:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T22:29:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ start_mirror cluster1:1 2024-04-24T22:29:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T22:29:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T22:29:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:29:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:1 2024-04-24T22:29:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T22:29:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T22:29:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T22:29:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T22:29:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster1:1 ']' 2024-04-24T22:29:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T22:29:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T22:29:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:29:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T22:29:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T22:29:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ start_mirror cluster1:2 2024-04-24T22:29:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T22:29:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T22:29:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:29:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:2 2024-04-24T22:29:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T22:29:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T22:29:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T22:29:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T22:29:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster1:2 ']' 2024-04-24T22:29:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T22:29:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T22:29:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:29:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T22:29:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T22:29:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ start_mirror cluster1:3 2024-04-24T22:29:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T22:29:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T22:29:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:29:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1:3 2024-04-24T22:29:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T22:29:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T22:29:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T22:29:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T22:29:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster1:3 ']' 2024-04-24T22:29:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T22:29:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T22:29:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:29:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T22:29:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T22:29:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ start_mirrors cluster2 2024-04-24T22:29:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:29:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T22:29:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ start_mirror cluster2:0 2024-04-24T22:29:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2:0 2024-04-24T22:29:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T22:29:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-04-24T22:29:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster2:0 2024-04-24T22:29:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T22:29:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T22:29:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster2 2024-04-24T22:29:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T22:29:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 = cluster2:0 ']' 2024-04-24T22:29:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster2 2024-04-24T22:29:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster2 2024-04-24T22:29:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T22:29:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:29:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T22:29:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T22:29:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ start_mirror cluster2:1 2024-04-24T22:29:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2:1 2024-04-24T22:29:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T22:29:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-04-24T22:29:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster2:1 2024-04-24T22:29:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T22:29:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T22:29:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster2 2024-04-24T22:29:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=1 2024-04-24T22:29:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 = cluster2:1 ']' 2024-04-24T22:29:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster2 2024-04-24T22:29:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster2 2024-04-24T22:29:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=1 2024-04-24T22:29:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:29:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T22:29:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T22:29:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ start_mirror cluster2:2 2024-04-24T22:29:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2:2 2024-04-24T22:29:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T22:29:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster2:2 cluster instance 2024-04-24T22:29:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster2:2 2024-04-24T22:29:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T22:29:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T22:29:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster2 2024-04-24T22:29:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=2 2024-04-24T22:29:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 2 = cluster2:2 ']' 2024-04-24T22:29:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster2 2024-04-24T22:29:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster2 2024-04-24T22:29:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=2 2024-04-24T22:29:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:29:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T22:29:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T22:29:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ start_mirror cluster2:3 2024-04-24T22:29:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2:3 2024-04-24T22:29:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T22:29:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster2:3 cluster instance 2024-04-24T22:29:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster2:3 2024-04-24T22:29:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T22:29:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T22:29:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster2 2024-04-24T22:29:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=3 2024-04-24T22:29:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 3 = cluster2:3 ']' 2024-04-24T22:29:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster2 2024-04-24T22:29:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster2 2024-04-24T22:29:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=3 2024-04-24T22:29:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:29:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T22:29:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T22:29:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-04-24T22:29:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ tee -a /tmp/tmp.DCvHjwkiaM/rbd-mirror.test.log 2024-04-24T22:29:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ date '+%F %T' 2024-04-24T22:29:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo 2024-04-24 22:29:31 TEST: add image and test replay after client crashes 2024-04-24T22:29:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:2024-04-24 22:29:31 TEST: add image and test replay after client crashes 2024-04-24T22:29:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=test 2024-04-24T22:29:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2024-04-24T22:29:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:29:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:29:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:29:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:29:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:29:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:29:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:29:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:29:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:29:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror test 512M 2024-04-24T22:29:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:29:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:29:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:29:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:29:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:29:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:29:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:29:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 512M ']' 2024-04-24T22:29:31.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=512M 2024-04-24T22:29:31.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:29:31.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/test 2024-04-24T22:29:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:29:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror test snapshot 2024-04-24T22:29:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:29:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:29:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:29:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:29:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2024-04-24T22:29:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:29:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/test 2024-04-24T22:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'test': 2024-04-24T22:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T22:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10b2bb4fcaaa 2024-04-24T22:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10b2bb4fcaaa 2024-04-24T22:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:29:31 2024 2024-04-24T22:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:29:31 2024 2024-04-24T22:29:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:29:31 2024 2024-04-24T22:29:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:29:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:29:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:29:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:29:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T22:29:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:29:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:29:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:29:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T22:29:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:29:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:29:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:29:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:29:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:29:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:29:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:29:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T22:29:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:29:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:29:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:29:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:29:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:29:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:29:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T22:29:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:29:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:29:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:29:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:29:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:29:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:29:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:29:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:29:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:29:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:29:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T22:29:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:29:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:29:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:29:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:29:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:29:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:29:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:29:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:29:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:29:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:29:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:29:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:29:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:29:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:29:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:29:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:29:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:29:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:29:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:29:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:29:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:29:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:29:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:29:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:29:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:29:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:29:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:29:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:29:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:29:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:29:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T22:29:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:29:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:29:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:29:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T22:29:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:29:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:29:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:29:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:29:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:29:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:29:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:29:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:29:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:29:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:29:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:29:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:29:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:29:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:29:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:29:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:29:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:29:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:29:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:29:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:29:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:29:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:29:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:29:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:29:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:29:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:29:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:29:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:29:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:29:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:29:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T22:29:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:29:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:29:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:29:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-24T22:29:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:29:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:29:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:29:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:29:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:29:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:29:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:29:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:29:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:29:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:29:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:29:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:29:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:29:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:29:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:29:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:29:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:29:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:29:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:29:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:29:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:29:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:29:33.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:29:33.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:29:33.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:29:33.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:29:33.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:29:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:29:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:29:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:29:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-24T22:29:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:29:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:29:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:29:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-24T22:29:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:29:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:29:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:29:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:29:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:29:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:29:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:29:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:29:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:29:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:29:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:29:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:29:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:29:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:29:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:29:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:29:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:29:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:29:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:29:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:29:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:29:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:29:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:29:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:29:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:29:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:29:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:29:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:29:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:29:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:29:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-24T22:29:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:29:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:29:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:29:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:29:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 1 2024-04-24T22:29:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:29:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:29:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T22:29:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:29:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:29:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:29:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:29:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:29:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:29:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:29:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:29:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:29:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:29:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:29:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:29:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:29:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:29:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:29:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:29:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:29:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:29:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:29:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:29:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:29:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:29:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:29:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:29:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:29:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:29:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:29:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:29:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:29:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:29:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T22:29:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:29:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:29:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:29:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T22:29:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:29:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:29:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:29:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:29:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:29:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:29:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:29:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:29:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:29:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:29:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:29:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:29:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:29:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:29:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:29:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:29:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:29:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:29:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:29:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:29:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:29:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:29:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:29:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:29:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:29:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:29:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:29:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:29:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:29:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:29:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T22:29:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:29:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:29:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:29:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-24T22:29:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:29:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:29:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:29:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:29:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:29:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:29:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:29:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:29:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:29:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:29:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:29:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:29:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:29:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:29:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:29:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:29:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:29:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:29:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:29:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:29:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:29:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:29:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:29:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:29:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:29:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:29:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:29:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:29:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:29:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:29:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-24T22:29:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:29:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:29:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:29:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-24T22:29:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:29:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:29:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:29:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:29:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:29:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:29:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:29:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:29:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:29:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:29:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:29:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:29:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:29:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:29:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:29:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:29:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:29:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:29:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:29:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:29:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:29:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:29:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:29:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:29:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:29:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:29:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:29:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:29:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:29:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:29:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-24T22:29:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:29:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:29:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:29:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:29:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 2 2024-04-24T22:29:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:29:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:29:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T22:29:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:29:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:29:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:29:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:29:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:29:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:29:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:29:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:29:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:29:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:29:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:29:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:29:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:29:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:29:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:29:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:29:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:29:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:29:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:29:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:29:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:29:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:29:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:29:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:29:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:29:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:29:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:29:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:29:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:29:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:29:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T22:29:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:29:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:29:37.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:29:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T22:29:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:29:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:29:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:29:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:29:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:29:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:29:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:29:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:29:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:29:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:29:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:29:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:29:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:29:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:29:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:29:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:29:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:29:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:29:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:29:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:29:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:29:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:29:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:29:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:29:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:29:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:29:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:29:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:29:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:29:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:29:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T22:29:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:29:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:29:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:29:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-24T22:29:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:29:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:29:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:29:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:29:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:29:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:29:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:29:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:29:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:29:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:29:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:29:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:29:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:29:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:29:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:29:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:29:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:29:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:29:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:29:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:29:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:29:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:29:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:29:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:29:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:29:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:29:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:29:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:29:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:29:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:29:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-24T22:29:37.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:29:37.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:29:37.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:29:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-24T22:29:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:29:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:29:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:29:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:29:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:29:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:29:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:29:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:29:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:29:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:29:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:29:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:29:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:29:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:29:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:29:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:29:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:29:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:29:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:29:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:29:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:29:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:29:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:29:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:29:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:29:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:29:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:29:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:29:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:29:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:29:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-24T22:29:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:29:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:29:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:29:37.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:29:37.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 4 2024-04-24T22:29:41.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:29:41.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:29:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T22:29:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:29:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:29:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:29:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:29:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:29:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:29:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:29:41.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:29:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:29:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:29:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:29:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:29:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:29:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:29:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:29:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:29:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:29:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:29:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:29:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:29:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:29:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:29:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:29:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:29:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:29:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:29:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:29:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:29:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:29:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:29:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T22:29:41.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:29:41.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:29:41.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:29:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T22:29:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:29:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:29:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:29:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:29:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:29:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:29:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:29:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:29:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:29:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:29:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:29:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:29:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:29:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:29:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:29:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:29:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:29:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:29:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:29:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:29:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:29:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:29:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:29:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:29:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:29:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:29:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:29:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:29:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:29:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:29:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T22:29:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:29:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:29:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:29:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-24T22:29:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:29:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:29:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:29:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:29:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:29:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:29:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:29:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:29:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:29:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:29:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:29:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:29:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:29:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:29:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:29:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:29:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:29:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:29:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:29:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:29:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:29:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:29:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:29:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:29:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:29:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:29:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:29:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:29:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:29:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:29:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-24T22:29:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:29:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:29:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:29:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-24T22:29:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:29:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:29:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:29:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:29:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:29:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:29:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:29:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:29:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:29:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:29:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:29:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:29:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:29:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:29:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:29:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:29:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:29:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:29:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:29:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:29:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:41.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:29:41.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:29:41.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:29:41.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:29:41.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:29:41.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:29:41.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:29:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:29:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:29:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:29:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-24T22:29:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:29:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:29:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:29:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:29:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 8 2024-04-24T22:29:47.353 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6506686381500988) > weight (0.25) 2024-04-24T22:29:47.353 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-04-24T22:29:47.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-04-24T22:29:47.354 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-04-24T22:29:47.354 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-04-24T22:29:47.354 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:29:47.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T22:29:47.354+0000 7ff3d72a9700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 47953) UID: 1000 2024-04-24T22:29:47.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:29:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:29:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:29:49.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T22:29:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:29:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:29:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:29:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:29:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:29:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:29:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:29:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:29:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:29:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:29:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:29:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:29:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:29:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:29:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:29:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:29:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:29:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:29:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:29:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:29:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:29:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:29:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:29:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:29:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:29:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:29:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:29:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:29:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:29:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:29:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T22:29:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:29:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:29:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:29:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T22:29:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:29:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:29:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:29:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:29:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:29:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:29:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:29:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:29:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:29:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:29:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:29:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:29:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:29:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:29:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:29:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:29:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:29:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:29:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:29:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:29:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:29:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:29:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:29:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:29:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:29:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:29:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:29:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:29:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:29:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:29:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T22:29:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T22:29:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-04-24T22:29:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:29:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:29:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:29:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:29:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:29:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:29:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:29:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:29:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:29:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:29:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:29:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:29:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:29:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:29:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:29:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:29:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:29:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:29:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:29:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:29:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:29:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:29:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:29:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:29:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:29:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:29:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:29:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:29:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:29:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:29:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-04-24T22:29:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:29:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:29:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:29:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-04-24T22:29:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:29:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:29:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:29:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:29:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:29:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:29:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:29:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:29:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:29:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:29:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:29:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:29:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:29:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:29:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:29:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:29:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:29:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:29:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:29:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:29:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:29:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:29:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:29:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:29:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:29:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:29:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:29:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:29:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:29:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:29:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-04-24T22:29:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:29:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:29:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:29:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:29:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 8 2024-04-24T22:29:53.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:29:53.456 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-04-24T22:29:53.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-24T22:29:53.457 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-24T22:29:53.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-24T22:29:53.460 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2024-04-24T22:29:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:29:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:29:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T22:29:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:29:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:29:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:29:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:29:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:29:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:29:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:29:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:29:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:29:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:29:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:29:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:29:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:29:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:29:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:29:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:29:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:29:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:29:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:29:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:29:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:29:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:29:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:29:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:29:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:29:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:29:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:29:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:29:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:29:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:29:58.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T22:29:58.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T22:29:58.315+0000 7ff750626840 -1 rbd::mirror::Mirror: 0x555d690b9560 update_pool_replayers: restarting failed pool replayer for uuid: a56abb4b-e2ab-4eb0-9450-fa5d0a242dd6 cluster: cluster2 client: client.mirror 2024-04-24T22:29:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:29:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:29:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:29:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:29:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T22:29:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:29:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:29:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:29:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:29:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:29:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:29:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:29:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:29:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:29:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:29:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:29:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:29:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:29:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:29:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:29:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:29:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:29:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:29:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:29:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:29:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:29:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:29:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:29:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:29:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:29:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:29:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:29:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:29:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:29:58.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T22:29:58.413+0000 7ff750626840 -1 rbd::mirror::Mirror: 0x555d690b9560 update_pool_replayers: restarting failed pool replayer for uuid: ebce59bf-8fae-4852-a495-05a331533a62 cluster: cluster2 client: client.mirror 2024-04-24T22:29:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:29:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:29:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:29:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T22:29:58.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T22:29:58.478+0000 7ff7e2e8a840 -1 rbd::mirror::Mirror: 0x55a7e6221560 update_pool_replayers: restarting failed pool replayer for uuid: a56abb4b-e2ab-4eb0-9450-fa5d0a242dd6 cluster: cluster2 client: client.mirror 2024-04-24T22:29:58.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T22:29:58.515+0000 7f4a66c6c840 -1 rbd::mirror::Mirror: 0x55dcfd55f560 update_pool_replayers: restarting failed pool replayer for uuid: a56abb4b-e2ab-4eb0-9450-fa5d0a242dd6 cluster: cluster2 client: client.mirror 2024-04-24T22:29:58.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T22:29:58.520+0000 7ff7e2e8a840 -1 rbd::mirror::Mirror: 0x55a7e6221560 update_pool_replayers: restarting failed pool replayer for uuid: ebce59bf-8fae-4852-a495-05a331533a62 cluster: cluster2 client: client.mirror 2024-04-24T22:29:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:29:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:29:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:29:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:29:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:29:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:29:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:29:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:29:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ clean_snap_name= 2024-04-24T22:29:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 10 2024-04-24T22:29:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:29:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror test 2024-04-24T22:29:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:29:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:29:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:29:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:29:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=7 2024-04-24T22:29:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:29:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-24T22:29:58.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T22:29:58.590+0000 7f4a66c6c840 -1 rbd::mirror::Mirror: 0x55dcfd55f560 update_pool_replayers: restarting failed pool replayer for uuid: ebce59bf-8fae-4852-a495-05a331533a62 cluster: cluster2 client: client.mirror 2024-04-24T22:29:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 604160 bytes 2024-04-24T22:29:58.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 30575616 bytes 2024-04-24T22:29:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 54102016 bytes 2024-04-24T22:29:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 101296128 bytes 2024-04-24T22:29:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 138763264 bytes 2024-04-24T22:29:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 191773696 bytes 2024-04-24T22:29:59.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 203565056 bytes 2024-04-24T22:29:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 228113408 bytes 2024-04-24T22:29:59.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 273395712 bytes 2024-04-24T22:29:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 315765760 bytes 2024-04-24T22:29:59.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 353311744 bytes 2024-04-24T22:29:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 477437952 bytes 2024-04-24T22:30:00.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 524852224 bytes 2024-04-24T22:30:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 581190656 bytes 2024-04-24T22:30:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 616531968 bytes 2024-04-24T22:30:00.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 658476032 bytes 2024-04-24T22:30:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 710095872 bytes 2024-04-24T22:30:00.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 741393408 bytes 2024-04-24T22:30:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 773807104 bytes 2024-04-24T22:30:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 797508608 bytes 2024-04-24T22:30:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 852257792 bytes 2024-04-24T22:30:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 878148608 bytes 2024-04-24T22:30:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 898510848 bytes 2024-04-24T22:30:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 934557696 bytes 2024-04-24T22:30:01.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 951616512 bytes 2024-04-24T22:30:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 995544064 bytes 2024-04-24T22:30:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1003665408 bytes 2024-04-24T22:30:01.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1010668544 bytes 2024-04-24T22:30:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1050350592 bytes 2024-04-24T22:30:01.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1074969600 bytes 2024-04-24T22:30:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1096310784 bytes 2024-04-24T22:30:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1214212096 bytes 2024-04-24T22:30:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1281125376 bytes 2024-04-24T22:30:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1286920192 bytes 2024-04-24T22:30:02.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1340963840 bytes 2024-04-24T22:30:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1375780864 bytes 2024-04-24T22:30:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1467553792 bytes 2024-04-24T22:30:02.353 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7132845666747648) > weight (0.25) 2024-04-24T22:30:02.353 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-04-24T22:30:02.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-04-24T22:30:02.354 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-04-24T22:30:02.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:30:02.353+0000 7efffa71e700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 48115) UID: 1000 2024-04-24T22:30:02.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1504354304 bytes 2024-04-24T22:30:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1579260928 bytes 2024-04-24T22:30:02.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:30:02.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1607930880 bytes 2024-04-24T22:30:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1664227328 bytes 2024-04-24T22:30:02.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1713217536 bytes 2024-04-24T22:30:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1757503488 bytes 2024-04-24T22:30:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1759349760 bytes 2024-04-24T22:30:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1794510848 bytes 2024-04-24T22:30:03.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1831769088 bytes 2024-04-24T22:30:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1873797120 bytes 2024-04-24T22:30:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1879119872 bytes 2024-04-24T22:30:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 1919388672 bytes 2024-04-24T22:30:03.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 1948512256 bytes 2024-04-24T22:30:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 1962422272 bytes 2024-04-24T22:30:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 1968860160 bytes 2024-04-24T22:30:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2026468352 bytes 2024-04-24T22:30:03.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2040638464 bytes 2024-04-24T22:30:03.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2080373760 bytes 2024-04-24T22:30:03.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2082742272 bytes 2024-04-24T22:30:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2136889344 bytes 2024-04-24T22:30:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2206310400 bytes 2024-04-24T22:30:04.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2265467904 bytes 2024-04-24T22:30:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2309848064 bytes 2024-04-24T22:30:04.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2322151424 bytes 2024-04-24T22:30:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2368435200 bytes 2024-04-24T22:30:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2444625920 bytes 2024-04-24T22:30:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2455326720 bytes 2024-04-24T22:30:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2518516736 bytes 2024-04-24T22:30:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2555961344 bytes 2024-04-24T22:30:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2610416640 bytes 2024-04-24T22:30:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2673636352 bytes 2024-04-24T22:30:05.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2722985984 bytes 2024-04-24T22:30:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 2807602176 bytes 2024-04-24T22:30:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:30:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:30:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:30:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:30:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-24T22:30:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:30:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:30:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:30:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:30:05.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:30:05.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:30:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T22:30:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:30:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:30:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:30:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:30:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:30:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:30:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T22:30:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T22:30:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:30:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: down+unknown 2024-04-24T22:30:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: status not found 2024-04-24T22:30:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24T22:30:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T22:30:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T22:30:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T22:30:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T22:30:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:30:00' 2024-04-24T22:30:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T22:30:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T22:30:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:30:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 2 2024-04-24T22:30:08.355 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-04-24T22:30:08.355 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:30:08.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:30:08.355 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-04-24T22:30:08.355 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-04-24T22:30:08.356 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-04-24T22:30:08.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-04-24T22:30:08.359 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2024-04-24T22:30:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T22:30:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:30:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:30:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:30:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:30:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:30:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:30:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T22:30:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T22:30:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:30:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: down+unknown 2024-04-24T22:30:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: status not found 2024-04-24T22:30:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24T22:30:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T22:30:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T22:30:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T22:30:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T22:30:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:30:00' 2024-04-24T22:30:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T22:30:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T22:30:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:30:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 4 2024-04-24T22:30:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T22:30:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:30:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:30:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:30:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:30:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:30:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:30:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T22:30:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T22:30:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:30:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: down+unknown 2024-04-24T22:30:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: status not found 2024-04-24T22:30:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24T22:30:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T22:30:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T22:30:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T22:30:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T22:30:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:30:00' 2024-04-24T22:30:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T22:30:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T22:30:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:30:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 8 2024-04-24T22:30:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T22:30:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:30:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:30:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:30:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:30:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:30:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:30:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T22:30:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T22:30:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:30:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: down+unknown 2024-04-24T22:30:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: status not found 2024-04-24T22:30:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24T22:30:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T22:30:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T22:30:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T22:30:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T22:30:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:30:00' 2024-04-24T22:30:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T22:30:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 1 2024-04-24T22:30:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:30:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 8 2024-04-24T22:30:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T22:30:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:30:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:30:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:30:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:30:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:30:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:30:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T22:30:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T22:30:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:30:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+replaying 2024-04-24T22:30:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713997772,"remote_snapshot_timestamp":1713997772,"replay_state":"idle"} 2024-04-24T22:30:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: service: mirror.1 on smithi134 2024-04-24T22:30:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:30:23 2024-04-24T22:30:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T22:30:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T22:30:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T22:30:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T22:30:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:30:28' 2024-04-24T22:30:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T22:30:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: state: up+replaying 2024-04-24T22:30:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ description: .*' 2024-04-24T22:30:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713997772,"remote_snapshot_timestamp":1713997772,"replay_state":"idle"} 2024-04-24T22:30:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:30:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T22:30:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^up+' 2024-04-24T22:30:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T22:30:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T22:30:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: service: mirror.1 on smithi134 2024-04-24T22:30:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update 2024-04-24T22:30:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T22:30:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 22:30:23' 2024-04-24T22:30:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-24 22:30:23' 2024-04-24T22:30:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:30:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:30:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'prev_last_update=2024-04-24 22:30:23' 2024-04-24T22:30:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:30:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:30:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T22:30:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='OKOKOK142044313mirror.0smithi13417.2.7-915-g330067dbfalseOK42124294mirror.2smithi13417.2.7-915-g330067dbfalseOK42164316mirror.3smithi13417.2.7-915-g330067dbfalseOK42804283mirror.1smithi13417.2.7-915-g330067dbtrueOKtest829c41fd-bfa9-46e0-a6ae-748aa71d3214up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713997772,"remote_snapshot_timestamp":1713997772,"replay_state":"idle"}42804283mirror.1smithi1342024-04-24 22:30:23cluster2d9b73256-b8ac-4751-9b60-95a0234fa37cup+stoppedlocal image is primary2024-04-24 22:30:28' 2024-04-24T22:30:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update state 2024-04-24T22:30:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-24T22:30:28.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 22:30:23' 2024-04-24T22:30:28.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-24T22:30:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=up+replaying 2024-04-24T22:30:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T22:30:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep up+replaying 2024-04-24T22:30:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T22:30:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:30:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:30:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap1 2024-04-24T22:30:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap1 2024-04-24T22:30:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:30:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap1 2024-04-24T22:30:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:30:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T22:30:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:30:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T22:30:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:30:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:30:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:30:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:30:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:30:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T22:30:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:30:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:30:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:30:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:30:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T22:30:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:30:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:30:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:30:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:30:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:30:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:30:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:30:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:30:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:30:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:30:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:30:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T22:30:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:30:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:30:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:30:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:30:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:30:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:30:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:30:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:30:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:30:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:30:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:30:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:30:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:30:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:30:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:30:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:30:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:30:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:30:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:30:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:30:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:30:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:30:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:30:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:30:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:30:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:30:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:30:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:30:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:30:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:30:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:30:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:30:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:30:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T22:30:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:30:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:30:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:30:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:30:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T22:30:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:30:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:30:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:30:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:30:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:30:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:30:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:30:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:30:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:30:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:30:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:30:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:30:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:30:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:30:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:30:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:30:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:30:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:30:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:30:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:30:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:30:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:30:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:30:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:30:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:30:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:30:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:30:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:30:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:30:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:30:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:30:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:30:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:30:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T22:30:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:30:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:30:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:30:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:30:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:30:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:30:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:30:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:30:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-24T22:30:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:30:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:30:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:30:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:30:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-24T22:30:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:30:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:30:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-test.status 2024-04-24T22:30:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-test.status 2024-04-24T22:30:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:30:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:30:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-test.status 2024-04-24T22:30:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-test.status 2024-04-24T22:30:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:30:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:30:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-24T22:30:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:30:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-24T22:30:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 6 2024-04-24T22:30:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:30:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:30:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:30:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-24T22:30:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:30:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:30:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=6 2024-04-24T22:30:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:30:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:30:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:30:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:30:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:30:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:30:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:30:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:30:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:30:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:30:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:30:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:30:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:30:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:30:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = '' 2024-04-24T22:30:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:30:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:30:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:30:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:30:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:30:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:30:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:30:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:30:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:30:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:30:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:30:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:30:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:30:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = '' 2024-04-24T22:30:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:30:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:30:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:30:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:30:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:30:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:30:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:30:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:30:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:30:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:30:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:30:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:30:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:30:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = '' 2024-04-24T22:30:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:30:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T22:30:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:30:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:30:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:41.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:30:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:30:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:30:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:30:41.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=6 2024-04-24T22:30:41.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 6 = 6 2024-04-24T22:30:41.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:30:41.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-04-24T22:30:41.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:30:41.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:41.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap1 2024-04-24T22:30:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:30:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:30:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:30:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-04-24T22:30:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'test': 2024-04-24T22:30:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T22:30:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:30:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:30:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10bb1f6f9c8b 2024-04-24T22:30:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10bb1f6f9c8b 2024-04-24T22:30:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:30:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:30:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:30:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:30:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:30:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:30:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:30:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:30:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:30:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:30:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:30:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:30:42.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:30:42.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:30:42.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap1 2024-04-24T22:30:42.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap1 2024-04-24T22:30:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:30:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:30:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:30:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:30:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:30:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:30:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:30:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:30:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:30:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:30:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:30:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ clean_snap_name=snap1-clean 2024-04-24T22:30:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-04-24T22:30:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:30:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap1-clean 2024-04-24T22:30:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-04-24T22:30:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:30:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror test 2024-04-24T22:30:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:30:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:30:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=13 2024-04-24T22:30:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:30:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-24T22:30:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 56544256 bytes 2024-04-24T22:30:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 101053440 bytes 2024-04-24T22:30:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 101668864 bytes 2024-04-24T22:30:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 132633600 bytes 2024-04-24T22:30:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 133240832 bytes 2024-04-24T22:30:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 186072064 bytes 2024-04-24T22:30:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 227678208 bytes 2024-04-24T22:30:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 251809792 bytes 2024-04-24T22:30:46.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 264080384 bytes 2024-04-24T22:30:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 289438720 bytes 2024-04-24T22:30:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 309436416 bytes 2024-04-24T22:30:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 318295040 bytes 2024-04-24T22:30:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 400328704 bytes 2024-04-24T22:30:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 426200064 bytes 2024-04-24T22:30:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 452464640 bytes 2024-04-24T22:30:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 510517248 bytes 2024-04-24T22:30:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 552199168 bytes 2024-04-24T22:30:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 680657920 bytes 2024-04-24T22:30:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 751879168 bytes 2024-04-24T22:30:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 803956736 bytes 2024-04-24T22:30:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 819493888 bytes 2024-04-24T22:30:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 899417088 bytes 2024-04-24T22:30:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 935677952 bytes 2024-04-24T22:30:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 989229056 bytes 2024-04-24T22:30:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1089027072 bytes 2024-04-24T22:30:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1199364096 bytes 2024-04-24T22:30:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1259115520 bytes 2024-04-24T22:30:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1305925632 bytes 2024-04-24T22:30:48.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1425117184 bytes 2024-04-24T22:30:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1462085632 bytes 2024-04-24T22:30:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1509591040 bytes 2024-04-24T22:30:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1542042624 bytes 2024-04-24T22:30:48.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1607349248 bytes 2024-04-24T22:30:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1607994368 bytes 2024-04-24T22:30:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1608620032 bytes 2024-04-24T22:30:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1660564480 bytes 2024-04-24T22:30:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1706836992 bytes 2024-04-24T22:30:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1762408448 bytes 2024-04-24T22:30:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1818071040 bytes 2024-04-24T22:30:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1832100864 bytes 2024-04-24T22:30:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1904854016 bytes 2024-04-24T22:30:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1961794560 bytes 2024-04-24T22:30:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 2002756608 bytes 2024-04-24T22:30:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 2026338304 bytes 2024-04-24T22:30:49.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 2067986432 bytes 2024-04-24T22:30:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 2136584192 bytes 2024-04-24T22:30:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 2192190464 bytes 2024-04-24T22:30:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2218600448 bytes 2024-04-24T22:30:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2252899328 bytes 2024-04-24T22:30:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2301017088 bytes 2024-04-24T22:30:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2399567872 bytes 2024-04-24T22:30:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2412632064 bytes 2024-04-24T22:30:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2413252608 bytes 2024-04-24T22:30:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2426764288 bytes 2024-04-24T22:30:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2482035712 bytes 2024-04-24T22:30:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2549833728 bytes 2024-04-24T22:30:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2564234240 bytes 2024-04-24T22:30:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2649974784 bytes 2024-04-24T22:30:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2674720768 bytes 2024-04-24T22:30:51.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2675324928 bytes 2024-04-24T22:30:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2732710912 bytes 2024-04-24T22:30:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2775404544 bytes 2024-04-24T22:30:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2817132544 bytes 2024-04-24T22:30:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2837978112 bytes 2024-04-24T22:30:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2881242112 bytes 2024-04-24T22:30:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2904965120 bytes 2024-04-24T22:30:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2971091968 bytes 2024-04-24T22:30:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 3001311232 bytes 2024-04-24T22:30:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 3051811840 bytes 2024-04-24T22:30:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 3060075520 bytes 2024-04-24T22:30:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 3095710720 bytes 2024-04-24T22:30:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 3176419328 bytes 2024-04-24T22:30:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 3229288448 bytes 2024-04-24T22:30:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 3290103808 bytes 2024-04-24T22:30:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 3387014144 bytes 2024-04-24T22:30:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 3442387968 bytes 2024-04-24T22:30:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 3464637440 bytes 2024-04-24T22:30:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 3474781184 bytes 2024-04-24T22:30:53.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7900 IOs, 3508750336 bytes 2024-04-24T22:30:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8000 IOs, 3531087872 bytes 2024-04-24T22:30:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8100 IOs, 3577363456 bytes 2024-04-24T22:30:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8200 IOs, 3649276928 bytes 2024-04-24T22:30:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8300 IOs, 3674474496 bytes 2024-04-24T22:30:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8400 IOs, 3720603648 bytes 2024-04-24T22:30:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8500 IOs, 3722498048 bytes 2024-04-24T22:30:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8600 IOs, 3731303424 bytes 2024-04-24T22:30:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8700 IOs, 3772423168 bytes 2024-04-24T22:30:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8800 IOs, 3835674624 bytes 2024-04-24T22:30:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8900 IOs, 3907972096 bytes 2024-04-24T22:30:54.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9000 IOs, 3991372800 bytes 2024-04-24T22:30:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9100 IOs, 4042078208 bytes 2024-04-24T22:30:54.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9200 IOs, 4120427520 bytes 2024-04-24T22:30:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9300 IOs, 4126451712 bytes 2024-04-24T22:30:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9400 IOs, 4130437120 bytes 2024-04-24T22:30:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9500 IOs, 4189069312 bytes 2024-04-24T22:30:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9600 IOs, 4204522496 bytes 2024-04-24T22:30:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9700 IOs, 4259970048 bytes 2024-04-24T22:30:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9800 IOs, 4338038784 bytes 2024-04-24T22:30:55.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9900 IOs, 4376029184 bytes 2024-04-24T22:30:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10000 IOs, 4404430848 bytes 2024-04-24T22:30:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10100 IOs, 4443153408 bytes 2024-04-24T22:30:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10200 IOs, 4451211264 bytes 2024-04-24T22:30:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10300 IOs, 4461001728 bytes 2024-04-24T22:30:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10400 IOs, 4504515584 bytes 2024-04-24T22:30:55.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10500 IOs, 4523465728 bytes 2024-04-24T22:30:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10600 IOs, 4566894592 bytes 2024-04-24T22:30:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10700 IOs, 4593657856 bytes 2024-04-24T22:30:56.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10800 IOs, 4620694528 bytes 2024-04-24T22:30:56.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10900 IOs, 4646000640 bytes 2024-04-24T22:30:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:11000 IOs, 4686499840 bytes 2024-04-24T22:30:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:11100 IOs, 4741007360 bytes 2024-04-24T22:30:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:11200 IOs, 4822106112 bytes 2024-04-24T22:30:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:11300 IOs, 4865396736 bytes 2024-04-24T22:30:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:11400 IOs, 4898248704 bytes 2024-04-24T22:30:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:11500 IOs, 4963698688 bytes 2024-04-24T22:30:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:11600 IOs, 4982418432 bytes 2024-04-24T22:30:56.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:11700 IOs, 5014199296 bytes 2024-04-24T22:30:57.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:11800 IOs, 5036456960 bytes 2024-04-24T22:30:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:11900 IOs, 5113094144 bytes 2024-04-24T22:30:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:12000 IOs, 5196230656 bytes 2024-04-24T22:30:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:12100 IOs, 5244607488 bytes 2024-04-24T22:30:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:12200 IOs, 5320568832 bytes 2024-04-24T22:30:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:12300 IOs, 5418144768 bytes 2024-04-24T22:30:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:12400 IOs, 5466939392 bytes 2024-04-24T22:30:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:12500 IOs, 5509519360 bytes 2024-04-24T22:30:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:12600 IOs, 5564999680 bytes 2024-04-24T22:30:58.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:30:58.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:30:58.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:30:58.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:30:58.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-24T22:30:58.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:30:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:30:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:30:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:30:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:30:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:30:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T22:30:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:30:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:30:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:30:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:30:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:30:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:30:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T22:30:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T22:30:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:30:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+replaying 2024-04-24T22:30:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: replaying, {"bytes_per_second":17777800.53,"bytes_per_snapshot":266667008.0,"last_snapshot_bytes":533334016,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713997838,"remote_snapshot_timestamp":1713997838,"replay_state":"idle"} 2024-04-24T22:30:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: service: mirror.1 on smithi134 2024-04-24T22:30:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:30:53 2024-04-24T22:30:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T22:30:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T22:30:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T22:30:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T22:30:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:30:58' 2024-04-24T22:30:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T22:30:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: state: up+replaying 2024-04-24T22:30:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ description: .*' 2024-04-24T22:30:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: description: replaying, {"bytes_per_second":17777800.53,"bytes_per_snapshot":266667008.0,"last_snapshot_bytes":533334016,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713997838,"remote_snapshot_timestamp":1713997838,"replay_state":"idle"} 2024-04-24T22:30:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:30:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T22:30:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^up+' 2024-04-24T22:30:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T22:30:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T22:30:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: service: mirror.1 on smithi134 2024-04-24T22:30:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update 2024-04-24T22:30:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T22:30:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 22:30:53' 2024-04-24T22:30:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-24 22:30:53' 2024-04-24T22:30:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:30:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:30:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'prev_last_update=2024-04-24 22:30:53' 2024-04-24T22:30:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:30:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:30:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T22:30:59.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='OKOKOK142044313mirror.0smithi13417.2.7-915-g330067dbfalseOK42124294mirror.2smithi13417.2.7-915-g330067dbfalseOK42164316mirror.3smithi13417.2.7-915-g330067dbfalseOK42804283mirror.1smithi13417.2.7-915-g330067dbtrueOKtest829c41fd-bfa9-46e0-a6ae-748aa71d3214up+replayingreplaying, {"bytes_per_second":17777800.53,"bytes_per_snapshot":266667008.0,"last_snapshot_bytes":533334016,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713997838,"remote_snapshot_timestamp":1713997838,"replay_state":"idle"}42804283mirror.1smithi1342024-04-24 22:30:53cluster2d9b73256-b8ac-4751-9b60-95a0234fa37cup+stoppedlocal image is primary2024-04-24 22:30:58' 2024-04-24T22:30:59.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update state 2024-04-24T22:30:59.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-24T22:30:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 22:30:53' 2024-04-24T22:30:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-24T22:30:59.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=up+replaying 2024-04-24T22:30:59.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T22:30:59.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep up+replaying 2024-04-24T22:30:59.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T22:30:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:30:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:30:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap2 2024-04-24T22:30:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap2 2024-04-24T22:30:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:30:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:30:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:30:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap2 2024-04-24T22:30:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:31:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T22:31:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:31:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:31:29.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:31:29.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T22:31:29.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:31:29.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:31:29.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:31:29.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:31:29.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:31:29.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:31:29.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:31:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T22:31:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:31:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:31:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:31:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:31:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:31:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:31:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T22:31:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:31:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:31:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:31:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:31:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:31:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:31:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:31:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:31:30.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:31:30.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:31:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:31:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T22:31:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:31:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:31:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:31:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:31:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:31:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:31:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:31:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:31:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:31:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:31:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:31:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:31:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:31:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:31:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:31:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:31:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:31:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:31:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:31:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:31:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:31:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:31:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:31:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:31:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:31:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:31:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:31:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:31:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:31:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:31:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:31:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:31:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:31:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T22:31:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:31:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:31:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:31:31.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:31:31.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T22:31:31.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:31:31.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:31:31.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:31:31.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:31:31.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:31:31.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:31:31.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:31:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:31:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:31:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:31:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:31:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:31:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:31:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:31:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:31:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:31:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:31:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:31:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:31:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:31:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:31:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:31:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:31:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:31:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:31:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:31:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:31:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:31:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:31:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:31:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:31:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:31:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:31:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T22:31:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:31:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:31:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:31:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:31:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:31:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:31:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:31:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:31:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-24T22:31:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:31:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:31:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:31:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:31:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:31:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:31:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-24T22:31:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:31:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:31:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:31:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:31:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-test.status 2024-04-24T22:31:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-test.status 2024-04-24T22:31:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:31:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:31:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-test.status 2024-04-24T22:31:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-test.status 2024-04-24T22:31:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:31:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:31:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-24T22:31:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:31:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:31:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:31:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-24T22:31:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 9 2024-04-24T22:31:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:31:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:31:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:31:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:31:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:31:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-24T22:31:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:31:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:31:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=9 2024-04-24T22:31:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:31:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:31:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:31:33.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:31:33.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:31:33.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:31:33.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:31:33.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:31:33.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:31:33.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:31:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:31:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:31:33.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:31:33.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:31:33.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:31:33.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:31:33.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = '' 2024-04-24T22:31:33.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:31:33.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:31:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:31:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:31:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:31:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:31:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:31:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:31:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:31:33.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:31:33.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:31:33.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:31:33.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:31:33.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:31:33.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:31:33.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = '' 2024-04-24T22:31:33.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:31:33.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:31:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:31:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:31:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:31:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:31:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:31:34.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:31:34.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:31:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:31:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:31:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:31:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:31:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:31:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:31:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = '' 2024-04-24T22:31:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:31:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T22:31:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:31:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:31:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:31:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:31:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:31:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:31:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:31:36.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:31:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=9 2024-04-24T22:31:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 9 = 9 2024-04-24T22:31:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:31:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-04-24T22:31:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:31:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:31:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:31:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap2 2024-04-24T22:31:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:31:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:31:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:31:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-04-24T22:31:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'test': 2024-04-24T22:31:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T22:31:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:31:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 4 2024-04-24T22:31:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10bb1f6f9c8b 2024-04-24T22:31:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10bb1f6f9c8b 2024-04-24T22:31:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:31:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:31:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:31:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:31:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:31:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:31:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:31:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:31:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:31:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:31:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:31:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:31:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:31:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snap1-clean ']' 2024-04-24T22:31:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap1-clean 2024-04-24T22:31:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:31:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:31:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap1-clean 2024-04-24T22:31:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:31:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:31:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:31:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:31:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:31:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:31:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:31:38.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:31:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:31:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:31:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:31:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap2 2024-04-24T22:31:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:31:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:31:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap2 2024-04-24T22:31:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:31:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:31:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:31:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:31:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:31:39.461 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6757495075684191) > weight (0.25) 2024-04-24T22:31:39.461 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.44038703370687116) > weight (0.25) 2024-04-24T22:31:39.461 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-04-24T22:31:39.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-04-24T22:31:39.462 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-04-24T22:31:39.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T22:31:39.462+0000 7ff7478e0700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 47969) UID: 1000 2024-04-24T22:31:39.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:31:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:31:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:31:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:31:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:31:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:31:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:31:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ clean_snap_name=snap2-clean 2024-04-24T22:31:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-04-24T22:31:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:31:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:31:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:31:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap2-clean 2024-04-24T22:31:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-04-24T22:31:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:31:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror test 2024-04-24T22:31:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:31:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:31:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:31:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:31:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=6 2024-04-24T22:31:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:31:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-24T22:31:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 6440960 bytes 2024-04-24T22:31:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 39130112 bytes 2024-04-24T22:31:41.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 100317184 bytes 2024-04-24T22:31:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 215948288 bytes 2024-04-24T22:31:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 235027456 bytes 2024-04-24T22:31:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 267822080 bytes 2024-04-24T22:31:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 361249792 bytes 2024-04-24T22:31:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 377019392 bytes 2024-04-24T22:31:42.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 421371904 bytes 2024-04-24T22:31:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 463644672 bytes 2024-04-24T22:31:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 464269312 bytes 2024-04-24T22:31:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 541049856 bytes 2024-04-24T22:31:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 542658560 bytes 2024-04-24T22:31:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 609666048 bytes 2024-04-24T22:31:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 623314944 bytes 2024-04-24T22:31:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 679656448 bytes 2024-04-24T22:31:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 735576064 bytes 2024-04-24T22:31:43.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 795364352 bytes 2024-04-24T22:31:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 813798400 bytes 2024-04-24T22:31:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 855683072 bytes 2024-04-24T22:31:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 856314880 bytes 2024-04-24T22:31:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 952294400 bytes 2024-04-24T22:31:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 988520448 bytes 2024-04-24T22:31:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 989161472 bytes 2024-04-24T22:31:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1045280768 bytes 2024-04-24T22:31:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1097249792 bytes 2024-04-24T22:31:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1133400064 bytes 2024-04-24T22:31:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1217724416 bytes 2024-04-24T22:31:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1228715008 bytes 2024-04-24T22:31:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1278405632 bytes 2024-04-24T22:31:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1345207296 bytes 2024-04-24T22:31:44.463 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-04-24T22:31:44.463 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:31:44.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:31:44.464 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-04-24T22:31:44.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-24T22:31:44.464 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-24T22:31:44.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-24T22:31:44.467 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2024-04-24T22:31:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1402915840 bytes 2024-04-24T22:31:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1445187584 bytes 2024-04-24T22:31:44.704 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:31:44.703+0000 7f0f4ef7b700 -1 reset not still connected to 0x561387a189c0 2024-04-24T22:31:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1488667648 bytes 2024-04-24T22:31:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1513757696 bytes 2024-04-24T22:31:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1597468672 bytes 2024-04-24T22:31:45.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1661387776 bytes 2024-04-24T22:31:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1679015936 bytes 2024-04-24T22:31:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1703049216 bytes 2024-04-24T22:31:45.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1816711168 bytes 2024-04-24T22:31:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1848770560 bytes 2024-04-24T22:31:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1849368576 bytes 2024-04-24T22:31:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1899008000 bytes 2024-04-24T22:31:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1912279040 bytes 2024-04-24T22:31:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1952217088 bytes 2024-04-24T22:31:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1992085504 bytes 2024-04-24T22:31:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 2026076160 bytes 2024-04-24T22:31:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2065632256 bytes 2024-04-24T22:31:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2120098816 bytes 2024-04-24T22:31:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2171529216 bytes 2024-04-24T22:31:46.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2196715520 bytes 2024-04-24T22:31:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2211563520 bytes 2024-04-24T22:31:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2252655616 bytes 2024-04-24T22:31:46.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2282816512 bytes 2024-04-24T22:31:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2320887808 bytes 2024-04-24T22:31:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2394778624 bytes 2024-04-24T22:31:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2422552576 bytes 2024-04-24T22:31:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:31:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:31:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:31:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:31:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-24T22:31:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:31:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:31:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:31:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:31:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:31:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:31:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:31:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:31:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T22:31:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:31:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:31:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:31:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:31:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:31:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:31:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:31:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:31:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T22:31:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T22:31:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:31:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+replaying 2024-04-24T22:31:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":266667008.0,"last_snapshot_bytes":533334016,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713997838,"remote_snapshot_timestamp":1713997838,"replay_state":"idle"} 2024-04-24T22:31:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: service: mirror.1 on smithi134 2024-04-24T22:31:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:31:23 2024-04-24T22:31:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T22:31:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T22:31:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T22:31:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T22:31:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:31:28' 2024-04-24T22:31:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T22:31:48.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: state: up+replaying 2024-04-24T22:31:48.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ description: .*' 2024-04-24T22:31:48.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":266667008.0,"last_snapshot_bytes":533334016,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713997838,"remote_snapshot_timestamp":1713997838,"replay_state":"idle"} 2024-04-24T22:31:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:31:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T22:31:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^up+' 2024-04-24T22:31:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T22:31:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T22:31:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: service: mirror.1 on smithi134 2024-04-24T22:31:48.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update 2024-04-24T22:31:48.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T22:31:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 22:31:23' 2024-04-24T22:31:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-24 22:31:23' 2024-04-24T22:31:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:31:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:31:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:31:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:31:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'prev_last_update=2024-04-24 22:31:23' 2024-04-24T22:31:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:31:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:31:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T22:31:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='WARNINGWARNINGOK142044313mirror.0smithi13417.2.7-915-g330067dbfalseOK42164316mirror.3smithi13417.2.7-915-g330067dbfalseOK42804283mirror.1smithi13417.2.7-915-g330067dbtrueOK4396mirror.2smithi13417.2.7-915-g330067dbfalseWARNINGnot reporting statustest829c41fd-bfa9-46e0-a6ae-748aa71d3214up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":266667008.0,"last_snapshot_bytes":533334016,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713997838,"remote_snapshot_timestamp":1713997838,"replay_state":"idle"}42804283mirror.1smithi1342024-04-24 22:31:23cluster2d9b73256-b8ac-4751-9b60-95a0234fa37cup+stoppedlocal image is primary2024-04-24 22:31:28' 2024-04-24T22:31:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update state 2024-04-24T22:31:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-24T22:31:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 22:31:23' 2024-04-24T22:31:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-24T22:31:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=up+replaying 2024-04-24T22:31:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T22:31:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep up+replaying 2024-04-24T22:31:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T22:31:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:31:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:31:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap3 2024-04-24T22:31:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap3 2024-04-24T22:31:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:31:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:31:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:31:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap3 2024-04-24T22:31:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:32:05.360 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.27466523517525987) > weight (0.25) 2024-04-24T22:32:05.360 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-04-24T22:32:05.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-04-24T22:32:05.360 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-04-24T22:32:05.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:32:05.363+0000 7f8b527d7700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 50182) UID: 1000 2024-04-24T22:32:05.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:32:06.361 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-04-24T22:32:06.361 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:32:06.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:32:06.361 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-04-24T22:32:06.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-04-24T22:32:06.362 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-04-24T22:32:06.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-04-24T22:32:06.364 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2024-04-24T22:32:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T22:32:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:32:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:32:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:32:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T22:32:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:32:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:32:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:32:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:32:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:32:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:32:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:32:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T22:32:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:32:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:32:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:32:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:32:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:32:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:32:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T22:32:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:32:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:32:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:32:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:32:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:32:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:32:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:32:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:32:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:32:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:32:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:32:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T22:32:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:32:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:32:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:32:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:32:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:32:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:32:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:32:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:32:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:32:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:32:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:32:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:32:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:32:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:32:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:32:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:32:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:32:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:32:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:32:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:32:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:32:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:32:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:32:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:32:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:32:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:32:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:32:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:32:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:32:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:32:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:32:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:32:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:32:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T22:32:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:32:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:32:19.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:32:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:32:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T22:32:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:32:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:32:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:32:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:32:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:32:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:32:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:32:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:32:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:32:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:32:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:32:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:32:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:32:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:32:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:32:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:32:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:32:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:32:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:32:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:32:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:32:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:32:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:32:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:32:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:32:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:32:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:32:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:32:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:32:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:32:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:32:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:32:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:32:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T22:32:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:32:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:32:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:32:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:32:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:32:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:32:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:32:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:32:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-24T22:32:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:32:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:32:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:32:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:32:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:32:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:32:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-24T22:32:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:32:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:32:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:32:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:32:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-test.status 2024-04-24T22:32:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-test.status 2024-04-24T22:32:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:32:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:32:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-test.status 2024-04-24T22:32:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-test.status 2024-04-24T22:32:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:32:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:32:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-24T22:32:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:32:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:32:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:32:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-24T22:32:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 13 2024-04-24T22:32:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:32:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:32:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:32:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:32:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:32:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:32:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-24T22:32:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:32:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=13 2024-04-24T22:32:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:32:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:32:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:32:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:32:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:32:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:32:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:32:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:32:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:32:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:32:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:32:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:32:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:32:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:32:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:32:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:32:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 13 = '' 2024-04-24T22:32:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:32:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:32:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:32:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:32:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:32:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:32:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:32:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:32:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:32:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:32:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:32:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:32:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:32:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:32:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:32:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 13 = '' 2024-04-24T22:32:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:32:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:32:22.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:32:22.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:32:22.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:32:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:32:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:32:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:32:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:32:22.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:32:22.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:32:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:32:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:32:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:32:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:32:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 13 = '' 2024-04-24T22:32:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:32:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T22:32:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:32:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:32:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:32:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:32:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:32:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:32:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:32:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:32:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=13 2024-04-24T22:32:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 13 = 13 2024-04-24T22:32:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:32:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-04-24T22:32:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:32:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:32:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:32:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap3 2024-04-24T22:32:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:32:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:32:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:32:25.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-04-24T22:32:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'test': 2024-04-24T22:32:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T22:32:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:32:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 6 2024-04-24T22:32:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10bb1f6f9c8b 2024-04-24T22:32:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10bb1f6f9c8b 2024-04-24T22:32:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:32:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:32:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:32:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:32:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:32:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:31:38 2024 2024-04-24T22:32:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:32:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:32:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:32:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:32:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:32:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:32:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:32:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snap2-clean ']' 2024-04-24T22:32:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap2-clean 2024-04-24T22:32:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:32:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:32:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap2-clean 2024-04-24T22:32:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:32:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:32:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:32:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:32:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:32:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:32:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:32:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:32:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:32:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:32:27.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:32:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap3 2024-04-24T22:32:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:32:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:32:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap3 2024-04-24T22:32:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:32:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:32:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:32:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:32:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:32:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:32:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:32:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:32:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:32:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:32:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:32:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ clean_snap_name=snap3-clean 2024-04-24T22:32:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-04-24T22:32:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:32:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:32:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:32:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap3-clean 2024-04-24T22:32:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-04-24T22:32:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:32:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror test 2024-04-24T22:32:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:32:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:32:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:32:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:32:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=9 2024-04-24T22:32:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:32:29.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-24T22:32:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 14058496 bytes 2024-04-24T22:32:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 51090432 bytes 2024-04-24T22:32:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 68305920 bytes 2024-04-24T22:32:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 130644992 bytes 2024-04-24T22:32:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 205388800 bytes 2024-04-24T22:32:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 290902016 bytes 2024-04-24T22:32:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 319832064 bytes 2024-04-24T22:32:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 361342976 bytes 2024-04-24T22:32:30.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 401796096 bytes 2024-04-24T22:32:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 455612416 bytes 2024-04-24T22:32:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 478994432 bytes 2024-04-24T22:32:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 504664064 bytes 2024-04-24T22:32:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 546480128 bytes 2024-04-24T22:32:30.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 598225920 bytes 2024-04-24T22:32:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 652010496 bytes 2024-04-24T22:32:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 683153408 bytes 2024-04-24T22:32:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 687738880 bytes 2024-04-24T22:32:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 723224576 bytes 2024-04-24T22:32:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 773742592 bytes 2024-04-24T22:32:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 800243712 bytes 2024-04-24T22:32:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 851820544 bytes 2024-04-24T22:32:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 944661504 bytes 2024-04-24T22:32:31.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 989652992 bytes 2024-04-24T22:32:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1074254848 bytes 2024-04-24T22:32:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1104440320 bytes 2024-04-24T22:32:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1142484992 bytes 2024-04-24T22:32:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1162795008 bytes 2024-04-24T22:32:32.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1181350912 bytes 2024-04-24T22:32:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1266379776 bytes 2024-04-24T22:32:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1351972864 bytes 2024-04-24T22:32:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1401922560 bytes 2024-04-24T22:32:32.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1450940416 bytes 2024-04-24T22:32:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1499038720 bytes 2024-04-24T22:32:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1499643904 bytes 2024-04-24T22:32:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1505586176 bytes 2024-04-24T22:32:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1605617664 bytes 2024-04-24T22:32:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1672197120 bytes 2024-04-24T22:32:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1715376128 bytes 2024-04-24T22:32:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1802463232 bytes 2024-04-24T22:32:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1837421568 bytes 2024-04-24T22:32:33.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1921276928 bytes 2024-04-24T22:32:33.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1985673216 bytes 2024-04-24T22:32:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 2042539008 bytes 2024-04-24T22:32:34.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 2056526848 bytes 2024-04-24T22:32:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 2064415744 bytes 2024-04-24T22:32:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 2082044928 bytes 2024-04-24T22:32:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 2196445184 bytes 2024-04-24T22:32:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2197075968 bytes 2024-04-24T22:32:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2272207872 bytes 2024-04-24T22:32:34.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2276782080 bytes 2024-04-24T22:32:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2278866944 bytes 2024-04-24T22:32:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2287376384 bytes 2024-04-24T22:32:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2353610752 bytes 2024-04-24T22:32:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2402023424 bytes 2024-04-24T22:32:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2434557952 bytes 2024-04-24T22:32:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2517255168 bytes 2024-04-24T22:32:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2563513344 bytes 2024-04-24T22:32:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2595748864 bytes 2024-04-24T22:32:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2596363264 bytes 2024-04-24T22:32:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2644056064 bytes 2024-04-24T22:32:35.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2688258048 bytes 2024-04-24T22:32:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2711778304 bytes 2024-04-24T22:32:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2763273216 bytes 2024-04-24T22:32:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2834791424 bytes 2024-04-24T22:32:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2835416064 bytes 2024-04-24T22:32:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2904781824 bytes 2024-04-24T22:32:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2905384960 bytes 2024-04-24T22:32:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2965847040 bytes 2024-04-24T22:32:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 3043117056 bytes 2024-04-24T22:32:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 3091838976 bytes 2024-04-24T22:32:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 3126756352 bytes 2024-04-24T22:32:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 3185258496 bytes 2024-04-24T22:32:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 3243719680 bytes 2024-04-24T22:32:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 3284561920 bytes 2024-04-24T22:32:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 3287743488 bytes 2024-04-24T22:32:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 3355808768 bytes 2024-04-24T22:32:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 3361040384 bytes 2024-04-24T22:32:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 3399102464 bytes 2024-04-24T22:32:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7900 IOs, 3422199808 bytes 2024-04-24T22:32:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8000 IOs, 3461831680 bytes 2024-04-24T22:32:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8100 IOs, 3496653824 bytes 2024-04-24T22:32:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8200 IOs, 3544823808 bytes 2024-04-24T22:32:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:32:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:32:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:32:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:32:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-24T22:32:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:32:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:32:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:32:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:32:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:32:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:32:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:32:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:32:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T22:32:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:32:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:32:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:32:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:32:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:32:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:32:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:32:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:32:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T22:32:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T22:32:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:32:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+replaying 2024-04-24T22:32:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: replaying, {"bytes_per_second":14060270.93,"bytes_per_snapshot":532883968.0,"last_snapshot_bytes":529073152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713997940,"remote_snapshot_timestamp":1713997940,"replay_state":"idle"} 2024-04-24T22:32:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: service: mirror.1 on smithi134 2024-04-24T22:32:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:32:23 2024-04-24T22:32:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T22:32:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T22:32:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T22:32:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T22:32:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:32:28' 2024-04-24T22:32:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T22:32:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: state: up+replaying 2024-04-24T22:32:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ description: .*' 2024-04-24T22:32:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: description: replaying, {"bytes_per_second":14060270.93,"bytes_per_snapshot":532883968.0,"last_snapshot_bytes":529073152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713997940,"remote_snapshot_timestamp":1713997940,"replay_state":"idle"} 2024-04-24T22:32:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:32:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T22:32:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^up+' 2024-04-24T22:32:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T22:32:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T22:32:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: service: mirror.1 on smithi134 2024-04-24T22:32:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update 2024-04-24T22:32:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T22:32:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 22:32:23' 2024-04-24T22:32:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-24 22:32:23' 2024-04-24T22:32:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:32:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:32:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:32:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:32:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'prev_last_update=2024-04-24 22:32:23' 2024-04-24T22:32:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:32:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:32:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T22:32:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='OKOKOK142044313mirror.0smithi13417.2.7-915-g330067dbfalseOK42164316mirror.3smithi13417.2.7-915-g330067dbfalseOK42804283mirror.1smithi13417.2.7-915-g330067dbtrueOK43964399mirror.2smithi13417.2.7-915-g330067dbfalseOKtest829c41fd-bfa9-46e0-a6ae-748aa71d3214up+replayingreplaying, {"bytes_per_second":14060270.93,"bytes_per_snapshot":532883968.0,"last_snapshot_bytes":529073152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713997940,"remote_snapshot_timestamp":1713997940,"replay_state":"idle"}42804283mirror.1smithi1342024-04-24 22:32:23cluster2d9b73256-b8ac-4751-9b60-95a0234fa37cup+stoppedlocal image is primary2024-04-24 22:32:28' 2024-04-24T22:32:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update state 2024-04-24T22:32:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-24T22:32:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 22:32:23' 2024-04-24T22:32:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-24T22:32:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=up+replaying 2024-04-24T22:32:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T22:32:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep up+replaying 2024-04-24T22:32:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T22:32:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:32:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:32:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap4 2024-04-24T22:32:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap4 2024-04-24T22:32:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:32:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:32:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:32:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap4 2024-04-24T22:32:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:33:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T22:33:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:33:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:33:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:33:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T22:33:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:33:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:33:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:33:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:33:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:33:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:33:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:33:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T22:33:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:33:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:33:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:33:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:33:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:33:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:33:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T22:33:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:33:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:33:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:33:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:33:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:33:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:33:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:33:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T22:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:33:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:33:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:33:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:33:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:33:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:33:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:33:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:33:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:33:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:33:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:33:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:33:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:33:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:33:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:33:10.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:33:10.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:33:10.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:33:10.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T22:33:10.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:33:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:33:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:33:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:33:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T22:33:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:33:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:33:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:33:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:33:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:33:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:33:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:33:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:33:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:33:10.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:33:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:33:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:33:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:33:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:33:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:33:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:33:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:33:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:33:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:33:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:33:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:33:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:33:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:33:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:33:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:33:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:33:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:33:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:33:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:33:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:33:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:33:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:33:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:33:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T22:33:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:33:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:33:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:33:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:33:10.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:33:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:33:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:33:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:33:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-24T22:33:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:33:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:33:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:33:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:33:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:33:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:33:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-24T22:33:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:33:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:33:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:33:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:33:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-test.status 2024-04-24T22:33:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-test.status 2024-04-24T22:33:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:33:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:33:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-test.status 2024-04-24T22:33:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-test.status 2024-04-24T22:33:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:33:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:33:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-24T22:33:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:33:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:33:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:33:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-24T22:33:12.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 17 2024-04-24T22:33:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:33:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:33:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:33:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:33:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:33:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-24T22:33:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:33:12.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:33:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=17 2024-04-24T22:33:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:33:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:33:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:33:12.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:33:12.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:33:12.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:33:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:33:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:33:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:33:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:33:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:33:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:33:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:33:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:33:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:33:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:33:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 17 = '' 2024-04-24T22:33:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:33:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:33:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:33:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:33:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:33:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:33:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:33:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:33:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:33:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:33:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:33:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:33:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:33:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:33:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:33:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 17 = '' 2024-04-24T22:33:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:33:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:33:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:33:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:33:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:33:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:33:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:33:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:33:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:33:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:33:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:33:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:33:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:33:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:33:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:33:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 17 = '' 2024-04-24T22:33:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:33:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T22:33:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:33:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:33:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:33:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:33:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:33:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:33:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:33:15.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:33:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=17 2024-04-24T22:33:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 17 = 17 2024-04-24T22:33:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:33:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-04-24T22:33:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:33:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:33:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:33:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap4 2024-04-24T22:33:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:33:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:33:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:33:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-04-24T22:33:16.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'test': 2024-04-24T22:33:16.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T22:33:16.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:33:16.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 8 2024-04-24T22:33:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10bb1f6f9c8b 2024-04-24T22:33:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10bb1f6f9c8b 2024-04-24T22:33:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:33:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:33:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:33:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:33:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:33:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:31:38 2024 2024-04-24T22:33:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:33:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:33:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:33:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:33:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:33:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:33:16.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:33:16.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snap3-clean ']' 2024-04-24T22:33:16.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap3-clean 2024-04-24T22:33:16.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:33:16.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:33:16.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap3-clean 2024-04-24T22:33:16.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:33:16.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:33:16.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:33:16.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:33:16.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:33:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:33:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:33:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:33:17.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:33:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:33:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:33:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap4 2024-04-24T22:33:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:33:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:33:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap4 2024-04-24T22:33:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:33:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:33:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:33:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:33:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:33:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:33:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:33:19.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:33:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:33:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:33:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:33:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ clean_snap_name=snap4-clean 2024-04-24T22:33:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-04-24T22:33:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:33:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:33:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:33:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap4-clean 2024-04-24T22:33:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-04-24T22:33:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:33:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror test 2024-04-24T22:33:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:33:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:33:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:33:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:33:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=11 2024-04-24T22:33:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:33:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-24T22:33:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 31016960 bytes 2024-04-24T22:33:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 88505344 bytes 2024-04-24T22:33:20.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 89124864 bytes 2024-04-24T22:33:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 116059136 bytes 2024-04-24T22:33:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 139150336 bytes 2024-04-24T22:33:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 230095872 bytes 2024-04-24T22:33:21.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 239152128 bytes 2024-04-24T22:33:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 277633024 bytes 2024-04-24T22:33:21.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 294520832 bytes 2024-04-24T22:33:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 295604224 bytes 2024-04-24T22:33:21.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 335299584 bytes 2024-04-24T22:33:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 382046208 bytes 2024-04-24T22:33:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 422381568 bytes 2024-04-24T22:33:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 506343424 bytes 2024-04-24T22:33:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 527930368 bytes 2024-04-24T22:33:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 613556224 bytes 2024-04-24T22:33:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 679239680 bytes 2024-04-24T22:33:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 691201024 bytes 2024-04-24T22:33:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 733238272 bytes 2024-04-24T22:33:22.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 739362816 bytes 2024-04-24T22:33:22.468 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3235879275468093) > weight (0.25) 2024-04-24T22:33:22.468 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.39367647218746604) > weight (0.25) 2024-04-24T22:33:22.468 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-04-24T22:33:22.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-04-24T22:33:22.469 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-04-24T22:33:22.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T22:33:22.469+0000 7fb1d180b700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 51144) UID: 1000 2024-04-24T22:33:22.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 779975680 bytes 2024-04-24T22:33:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 839201792 bytes 2024-04-24T22:33:22.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:33:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 890586112 bytes 2024-04-24T22:33:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 931910656 bytes 2024-04-24T22:33:22.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 958572544 bytes 2024-04-24T22:33:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1008676864 bytes 2024-04-24T22:33:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1027611648 bytes 2024-04-24T22:33:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1082743808 bytes 2024-04-24T22:33:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1095411712 bytes 2024-04-24T22:33:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1121374208 bytes 2024-04-24T22:33:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1183197184 bytes 2024-04-24T22:33:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1231794176 bytes 2024-04-24T22:33:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1257093120 bytes 2024-04-24T22:33:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1331450880 bytes 2024-04-24T22:33:24.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1358370816 bytes 2024-04-24T22:33:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1411814400 bytes 2024-04-24T22:33:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1441266688 bytes 2024-04-24T22:33:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1498553344 bytes 2024-04-24T22:33:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1505211392 bytes 2024-04-24T22:33:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1547971584 bytes 2024-04-24T22:33:24.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1610411008 bytes 2024-04-24T22:33:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1708408832 bytes 2024-04-24T22:33:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1752723456 bytes 2024-04-24T22:33:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1754668032 bytes 2024-04-24T22:33:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1783226368 bytes 2024-04-24T22:33:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1827332096 bytes 2024-04-24T22:33:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1919352832 bytes 2024-04-24T22:33:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1951799296 bytes 2024-04-24T22:33:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 1981701120 bytes 2024-04-24T22:33:25.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2014931968 bytes 2024-04-24T22:33:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2024861696 bytes 2024-04-24T22:33:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2040922112 bytes 2024-04-24T22:33:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2051419136 bytes 2024-04-24T22:33:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2122341376 bytes 2024-04-24T22:33:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2176507904 bytes 2024-04-24T22:33:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2191864832 bytes 2024-04-24T22:33:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2242529280 bytes 2024-04-24T22:33:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2263873536 bytes 2024-04-24T22:33:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2330878976 bytes 2024-04-24T22:33:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2402057216 bytes 2024-04-24T22:33:26.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2450189312 bytes 2024-04-24T22:33:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2483792896 bytes 2024-04-24T22:33:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2530485248 bytes 2024-04-24T22:33:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2604120064 bytes 2024-04-24T22:33:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2646832128 bytes 2024-04-24T22:33:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2705658880 bytes 2024-04-24T22:33:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2746918912 bytes 2024-04-24T22:33:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2752619520 bytes 2024-04-24T22:33:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2779815936 bytes 2024-04-24T22:33:27.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 2840522752 bytes 2024-04-24T22:33:27.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 2848804864 bytes 2024-04-24T22:33:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 2936856576 bytes 2024-04-24T22:33:28.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 3018430464 bytes 2024-04-24T22:33:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 3070305280 bytes 2024-04-24T22:33:28.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 3111451648 bytes 2024-04-24T22:33:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 3182536704 bytes 2024-04-24T22:33:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 3269391360 bytes 2024-04-24T22:33:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 3322297344 bytes 2024-04-24T22:33:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7900 IOs, 3325680640 bytes 2024-04-24T22:33:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8000 IOs, 3349044224 bytes 2024-04-24T22:33:29.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8100 IOs, 3402180608 bytes 2024-04-24T22:33:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8200 IOs, 3406429184 bytes 2024-04-24T22:33:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8300 IOs, 3436421120 bytes 2024-04-24T22:33:29.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8400 IOs, 3465172992 bytes 2024-04-24T22:33:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8500 IOs, 3609209856 bytes 2024-04-24T22:33:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8600 IOs, 3649681408 bytes 2024-04-24T22:33:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8700 IOs, 3669200896 bytes 2024-04-24T22:33:29.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8800 IOs, 3741932544 bytes 2024-04-24T22:33:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8900 IOs, 3793217536 bytes 2024-04-24T22:33:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9000 IOs, 3814473728 bytes 2024-04-24T22:33:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9100 IOs, 3827795968 bytes 2024-04-24T22:33:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9200 IOs, 3878392832 bytes 2024-04-24T22:33:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9300 IOs, 3916864512 bytes 2024-04-24T22:33:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9400 IOs, 3917446144 bytes 2024-04-24T22:33:30.470 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-04-24T22:33:30.470 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:33:30.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:33:30.470 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-04-24T22:33:30.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-24T22:33:30.471 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-24T22:33:30.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-24T22:33:30.473 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2024-04-24T22:33:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9500 IOs, 3921751040 bytes 2024-04-24T22:33:30.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9600 IOs, 3928382464 bytes 2024-04-24T22:33:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9700 IOs, 3998053376 bytes 2024-04-24T22:33:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9800 IOs, 4046752768 bytes 2024-04-24T22:33:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9900 IOs, 4120978432 bytes 2024-04-24T22:33:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10000 IOs, 4132794368 bytes 2024-04-24T22:33:31.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:33:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:33:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:33:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:33:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-24T22:33:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:33:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:33:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:33:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:33:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:33:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:33:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:33:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:33:32.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T22:33:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:33:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:33:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:33:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:33:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:33:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:33:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:33:32.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:33:32.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T22:33:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T22:33:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:33:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+replaying 2024-04-24T22:33:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: replaying, {"bytes_per_second":17835349.33,"bytes_per_snapshot":532066816.0,"last_snapshot_bytes":535060480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713997991,"remote_snapshot_timestamp":1713997991,"replay_state":"idle"} 2024-04-24T22:33:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: service: mirror.1 on smithi134 2024-04-24T22:33:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:33:23 2024-04-24T22:33:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T22:33:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T22:33:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T22:33:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T22:33:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:33:28' 2024-04-24T22:33:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T22:33:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: state: up+replaying 2024-04-24T22:33:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ description: .*' 2024-04-24T22:33:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: description: replaying, {"bytes_per_second":17835349.33,"bytes_per_snapshot":532066816.0,"last_snapshot_bytes":535060480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713997991,"remote_snapshot_timestamp":1713997991,"replay_state":"idle"} 2024-04-24T22:33:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:33:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T22:33:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^up+' 2024-04-24T22:33:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T22:33:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T22:33:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: service: mirror.1 on smithi134 2024-04-24T22:33:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update 2024-04-24T22:33:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T22:33:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 22:33:23' 2024-04-24T22:33:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-24 22:33:23' 2024-04-24T22:33:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:33:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:33:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:33:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:33:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'prev_last_update=2024-04-24 22:33:23' 2024-04-24T22:33:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:33:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:33:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T22:33:32.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='WARNINGWARNINGOK142044313mirror.0smithi13417.2.7-915-g330067dbfalseOK42164316mirror.3smithi13417.2.7-915-g330067dbfalseOK42804283mirror.1smithi13417.2.7-915-g330067dbtrueOK4465mirror.2smithi13417.2.7-915-g330067dbfalseWARNINGnot reporting statustest829c41fd-bfa9-46e0-a6ae-748aa71d3214up+replayingreplaying, {"bytes_per_second":17835349.33,"bytes_per_snapshot":532066816.0,"last_snapshot_bytes":535060480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713997991,"remote_snapshot_timestamp":1713997991,"replay_state":"idle"}42804283mirror.1smithi1342024-04-24 22:33:23cluster2d9b73256-b8ac-4751-9b60-95a0234fa37cup+stoppedlocal image is primary2024-04-24 22:33:28' 2024-04-24T22:33:32.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update state 2024-04-24T22:33:32.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-24T22:33:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 22:33:23' 2024-04-24T22:33:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-24T22:33:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=up+replaying 2024-04-24T22:33:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T22:33:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep up+replaying 2024-04-24T22:33:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T22:33:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:33:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:33:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap5 2024-04-24T22:33:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap5 2024-04-24T22:33:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:33:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:33:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:33:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap5 2024-04-24T22:33:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:33:59.365 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3826891158968646) > weight (0.25) 2024-04-24T22:33:59.366 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7431653703738621) > weight (0.25) 2024-04-24T22:33:59.366 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7016144993476229) > weight (0.25) 2024-04-24T22:33:59.366 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-04-24T22:33:59.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-04-24T22:33:59.366 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-04-24T22:33:59.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T22:33:59.366+0000 7f8af0dfd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 48199) UID: 1000 2024-04-24T22:33:59.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:34:01.367 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-04-24T22:34:01.368 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:34:01.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:34:01.368 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-04-24T22:34:01.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-04-24T22:34:01.368 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-04-24T22:34:01.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-04-24T22:34:01.371 INFO:tasks.rbd_mirror_thrash:waiting for 92 secs before thrashing 2024-04-24T22:34:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T22:34:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:34:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T22:34:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:34:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:34:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:34:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:34:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:34:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T22:34:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:34:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:34:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:34:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:34:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T22:34:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:34:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:34:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:34:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:34:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:34:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:34:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:34:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:34:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:34:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:34:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:34:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T22:34:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:34:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:34:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:34:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:34:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:34:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:34:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:34:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:34:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:34:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:34:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:34:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:34:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:34:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:34:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:34:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:34:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:34:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:34:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:34:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:34:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:34:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:34:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:34:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:34:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:34:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:34:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:34:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:34:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:34:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:34:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:34:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:34:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:34:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T22:34:03.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:34:03.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:34:03.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:34:03.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:34:03.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T22:34:03.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:34:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:34:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:34:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:34:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:34:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:34:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:34:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:34:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:34:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:34:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:34:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:34:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:34:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:34:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:34:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:34:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:34:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:34:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:34:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:34:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:34:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:34:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:34:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:34:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:34:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:34:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:34:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:34:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:34:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:34:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:34:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:34:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:34:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T22:34:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:34:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:34:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:34:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:34:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:34:03.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:34:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:34:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:34:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-24T22:34:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:34:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:34:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:34:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:34:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-24T22:34:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:34:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:34:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:03.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-test.status 2024-04-24T22:34:03.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-test.status 2024-04-24T22:34:03.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:34:03.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:34:03.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-test.status 2024-04-24T22:34:03.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-test.status 2024-04-24T22:34:03.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:34:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:34:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-24T22:34:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:34:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-24T22:34:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 21 2024-04-24T22:34:05.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:34:05.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:34:05.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:05.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:05.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:34:05.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-24T22:34:05.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:34:05.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:34:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=21 2024-04-24T22:34:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:34:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:34:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:34:05.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:34:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:34:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:34:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:34:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:34:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:34:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:34:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:34:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:34:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:34:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:34:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 21 = '' 2024-04-24T22:34:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:34:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:34:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:34:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:34:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:34:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:34:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:34:05.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:34:05.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:34:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:34:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:34:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:34:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:34:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 21 = '' 2024-04-24T22:34:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:34:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:34:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:34:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:34:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:34:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:34:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:34:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:34:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:34:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:34:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:34:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:34:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:34:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 21 = '' 2024-04-24T22:34:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:34:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T22:34:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:34:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:34:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:34:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:34:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:34:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:34:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=21 2024-04-24T22:34:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 21 = 21 2024-04-24T22:34:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:34:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-04-24T22:34:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:34:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap5 2024-04-24T22:34:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:34:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:34:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:34:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-04-24T22:34:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'test': 2024-04-24T22:34:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T22:34:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:34:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 10 2024-04-24T22:34:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10bb1f6f9c8b 2024-04-24T22:34:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10bb1f6f9c8b 2024-04-24T22:34:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:34:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:34:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:34:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:34:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:34:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:33:17 2024 2024-04-24T22:34:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:34:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:34:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:34:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:34:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:34:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:34:09.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:34:09.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snap4-clean ']' 2024-04-24T22:34:09.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap4-clean 2024-04-24T22:34:09.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap4-clean 2024-04-24T22:34:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:34:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:34:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:34:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:34:09.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:34:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:34:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:34:10.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:34:10.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:34:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:34:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:34:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap5 2024-04-24T22:34:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap5 2024-04-24T22:34:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:34:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:34:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:34:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:34:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:34:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:34:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:34:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:34:12.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:34:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:34:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:34:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ clean_snap_name=snap5-clean 2024-04-24T22:34:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-04-24T22:34:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:34:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap5-clean 2024-04-24T22:34:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-04-24T22:34:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:34:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror test 2024-04-24T22:34:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:34:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:34:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=8 2024-04-24T22:34:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:34:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-24T22:34:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 56638464 bytes 2024-04-24T22:34:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 114166784 bytes 2024-04-24T22:34:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 178142208 bytes 2024-04-24T22:34:13.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 187040768 bytes 2024-04-24T22:34:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 248002560 bytes 2024-04-24T22:34:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 305075200 bytes 2024-04-24T22:34:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 344229888 bytes 2024-04-24T22:34:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 359005184 bytes 2024-04-24T22:34:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 406709248 bytes 2024-04-24T22:34:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 407312384 bytes 2024-04-24T22:34:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 441177088 bytes 2024-04-24T22:34:14.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 471655424 bytes 2024-04-24T22:34:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 483001344 bytes 2024-04-24T22:34:14.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 541488128 bytes 2024-04-24T22:34:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 579605504 bytes 2024-04-24T22:34:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 676391936 bytes 2024-04-24T22:34:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 719279104 bytes 2024-04-24T22:34:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 749558784 bytes 2024-04-24T22:34:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 796374016 bytes 2024-04-24T22:34:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 830039040 bytes 2024-04-24T22:34:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 847397888 bytes 2024-04-24T22:34:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 923483136 bytes 2024-04-24T22:34:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 948224000 bytes 2024-04-24T22:34:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1019520000 bytes 2024-04-24T22:34:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1032848384 bytes 2024-04-24T22:34:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1033465856 bytes 2024-04-24T22:34:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1065863168 bytes 2024-04-24T22:34:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1117005824 bytes 2024-04-24T22:34:16.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1155335168 bytes 2024-04-24T22:34:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1223330816 bytes 2024-04-24T22:34:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1304435712 bytes 2024-04-24T22:34:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1327768576 bytes 2024-04-24T22:34:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1381054464 bytes 2024-04-24T22:34:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1463941120 bytes 2024-04-24T22:34:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1515913216 bytes 2024-04-24T22:34:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1555294208 bytes 2024-04-24T22:34:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1607254016 bytes 2024-04-24T22:34:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1607872512 bytes 2024-04-24T22:34:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1626832896 bytes 2024-04-24T22:34:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1631167488 bytes 2024-04-24T22:34:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1645579264 bytes 2024-04-24T22:34:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1680073728 bytes 2024-04-24T22:34:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1720385536 bytes 2024-04-24T22:34:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1743883264 bytes 2024-04-24T22:34:18.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1802229760 bytes 2024-04-24T22:34:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1878128640 bytes 2024-04-24T22:34:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1971716096 bytes 2024-04-24T22:34:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2028237824 bytes 2024-04-24T22:34:18.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2060581888 bytes 2024-04-24T22:34:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2156574720 bytes 2024-04-24T22:34:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2200504320 bytes 2024-04-24T22:34:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2227805184 bytes 2024-04-24T22:34:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2251203584 bytes 2024-04-24T22:34:19.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2276398080 bytes 2024-04-24T22:34:19.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2394814464 bytes 2024-04-24T22:34:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2433133568 bytes 2024-04-24T22:34:19.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2524636160 bytes 2024-04-24T22:34:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2567502848 bytes 2024-04-24T22:34:19.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2609539072 bytes 2024-04-24T22:34:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2652706816 bytes 2024-04-24T22:34:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2684967936 bytes 2024-04-24T22:34:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2747158528 bytes 2024-04-24T22:34:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2765985792 bytes 2024-04-24T22:34:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2791449600 bytes 2024-04-24T22:34:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2854633472 bytes 2024-04-24T22:34:20.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2905155584 bytes 2024-04-24T22:34:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2999293952 bytes 2024-04-24T22:34:20.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 3033072640 bytes 2024-04-24T22:34:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 3122417664 bytes 2024-04-24T22:34:21.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:34:21.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:34:21.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:34:21.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:34:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-24T22:34:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:34:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:34:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:34:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:34:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:34:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:34:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T22:34:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:34:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:34:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:34:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:34:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:34:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:34:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T22:34:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T22:34:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:34:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+replaying 2024-04-24T22:34:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":532066816.0,"last_snapshot_bytes":535060480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713997991,"remote_snapshot_timestamp":1713997991,"replay_state":"idle"} 2024-04-24T22:34:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: service: mirror.1 on smithi134 2024-04-24T22:34:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:33:53 2024-04-24T22:34:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T22:34:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T22:34:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T22:34:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T22:34:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:33:58' 2024-04-24T22:34:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T22:34:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: state: up+replaying 2024-04-24T22:34:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ description: .*' 2024-04-24T22:34:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":532066816.0,"last_snapshot_bytes":535060480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713997991,"remote_snapshot_timestamp":1713997991,"replay_state":"idle"} 2024-04-24T22:34:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:34:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T22:34:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^up+' 2024-04-24T22:34:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T22:34:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T22:34:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: service: mirror.1 on smithi134 2024-04-24T22:34:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update 2024-04-24T22:34:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T22:34:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 22:33:53' 2024-04-24T22:34:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-24 22:33:53' 2024-04-24T22:34:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:34:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:34:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'prev_last_update=2024-04-24 22:33:53' 2024-04-24T22:34:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:34:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:34:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T22:34:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='OKOKOK142044313mirror.0smithi13417.2.7-915-g330067dbfalseOK42164316mirror.3smithi13417.2.7-915-g330067dbfalseOK42804283mirror.1smithi13417.2.7-915-g330067dbtrueOK44654468mirror.2smithi13417.2.7-915-g330067dbfalseOKtest829c41fd-bfa9-46e0-a6ae-748aa71d3214up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":532066816.0,"last_snapshot_bytes":535060480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713997991,"remote_snapshot_timestamp":1713997991,"replay_state":"idle"}42804283mirror.1smithi1342024-04-24 22:33:53cluster2d9b73256-b8ac-4751-9b60-95a0234fa37cup+stoppedlocal image is primary2024-04-24 22:33:58' 2024-04-24T22:34:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update state 2024-04-24T22:34:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-24T22:34:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 22:33:53' 2024-04-24T22:34:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-24T22:34:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=up+replaying 2024-04-24T22:34:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T22:34:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep up+replaying 2024-04-24T22:34:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T22:34:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:34:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:34:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap6 2024-04-24T22:34:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap6 2024-04-24T22:34:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:34:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap6 2024-04-24T22:34:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:34:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T22:34:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:34:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T22:34:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:34:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:34:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:34:50.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:34:50.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:34:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T22:34:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:34:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:34:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:34:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:34:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T22:34:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:34:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:34:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:34:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:34:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:34:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:34:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:34:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:34:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:34:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:34:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:34:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T22:34:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:34:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:34:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:34:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:34:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:34:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:34:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:34:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:34:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:34:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:34:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:34:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:34:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:34:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:34:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:34:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:34:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:34:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:34:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:34:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:34:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:34:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:34:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:34:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:34:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:34:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:34:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:34:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:34:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:34:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:34:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:34:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:34:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:34:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T22:34:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:34:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:34:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:34:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:34:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T22:34:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:34:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:34:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:34:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:34:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:34:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:34:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:34:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:34:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:34:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:34:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:34:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:34:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:34:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:34:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:34:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:34:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:34:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:34:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:34:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:34:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:34:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:34:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:34:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:34:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:34:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:34:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:34:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:34:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:34:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:34:51.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:34:51.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:34:51.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:34:51.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T22:34:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:34:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:34:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:34:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:34:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:34:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:34:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:34:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:34:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-24T22:34:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:34:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:34:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:34:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:34:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-24T22:34:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:34:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:34:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-test.status 2024-04-24T22:34:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-test.status 2024-04-24T22:34:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:34:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:34:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-test.status 2024-04-24T22:34:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-test.status 2024-04-24T22:34:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:34:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:34:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-24T22:34:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:34:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-24T22:34:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 25 2024-04-24T22:34:53.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:34:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:34:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:34:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:34:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-24T22:34:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:34:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=25 2024-04-24T22:34:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:34:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:34:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:34:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:34:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:34:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:34:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:34:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:34:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:34:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:34:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:34:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:34:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:34:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:34:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 25 = '' 2024-04-24T22:34:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:34:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:34:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:34:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:34:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:34:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:34:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:34:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:34:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:34:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:34:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:34:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:34:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:34:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 25 = '' 2024-04-24T22:34:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:34:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:34:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:34:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:34:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:34:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:34:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:34:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:34:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:34:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:34:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:34:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:34:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:34:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 25 = '' 2024-04-24T22:34:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:34:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T22:34:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:34:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:34:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:34:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:34:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:34:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:34:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=25 2024-04-24T22:34:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 25 = 25 2024-04-24T22:34:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:34:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-04-24T22:34:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:34:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap6 2024-04-24T22:34:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:34:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:34:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:34:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-04-24T22:34:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'test': 2024-04-24T22:34:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T22:34:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:34:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 12 2024-04-24T22:34:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10bb1f6f9c8b 2024-04-24T22:34:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10bb1f6f9c8b 2024-04-24T22:34:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:34:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:34:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:34:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:34:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:34:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:33:17 2024 2024-04-24T22:34:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:34:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:34:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:34:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:34:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:34:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:34:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:34:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snap5-clean ']' 2024-04-24T22:34:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap5-clean 2024-04-24T22:34:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap5-clean 2024-04-24T22:34:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:34:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:34:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:34:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:34:57.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:34:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:34:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:34:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:34:58.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:34:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:34:59.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:34:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap6 2024-04-24T22:34:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:34:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:34:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap6 2024-04-24T22:34:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:34:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:34:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:34:59.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:34:59.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:34:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:34:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:35:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:35:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:35:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:35:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:35:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ clean_snap_name=snap6-clean 2024-04-24T22:35:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-04-24T22:35:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:35:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap6-clean 2024-04-24T22:35:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-04-24T22:35:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:35:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror test 2024-04-24T22:35:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:35:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:35:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=6 2024-04-24T22:35:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:35:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-24T22:35:01.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 22732800 bytes 2024-04-24T22:35:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 45410304 bytes 2024-04-24T22:35:01.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 81247232 bytes 2024-04-24T22:35:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 107678720 bytes 2024-04-24T22:35:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 116381696 bytes 2024-04-24T22:35:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 231924736 bytes 2024-04-24T22:35:02.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 248042496 bytes 2024-04-24T22:35:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 328715264 bytes 2024-04-24T22:35:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 398635008 bytes 2024-04-24T22:35:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 422158336 bytes 2024-04-24T22:35:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 422759424 bytes 2024-04-24T22:35:02.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 439172096 bytes 2024-04-24T22:35:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 473187328 bytes 2024-04-24T22:35:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 483087360 bytes 2024-04-24T22:35:02.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 531812352 bytes 2024-04-24T22:35:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 597682176 bytes 2024-04-24T22:35:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 640094208 bytes 2024-04-24T22:35:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 696954880 bytes 2024-04-24T22:35:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 697579520 bytes 2024-04-24T22:35:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 789956608 bytes 2024-04-24T22:35:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 790553600 bytes 2024-04-24T22:35:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 830133248 bytes 2024-04-24T22:35:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 845186048 bytes 2024-04-24T22:35:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 846322688 bytes 2024-04-24T22:35:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 856924160 bytes 2024-04-24T22:35:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 889695232 bytes 2024-04-24T22:35:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 908966912 bytes 2024-04-24T22:35:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 951317504 bytes 2024-04-24T22:35:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1004035072 bytes 2024-04-24T22:35:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1029055488 bytes 2024-04-24T22:35:04.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1077641216 bytes 2024-04-24T22:35:05.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1150517248 bytes 2024-04-24T22:35:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1180106752 bytes 2024-04-24T22:35:05.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1225479168 bytes 2024-04-24T22:35:05.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1261243392 bytes 2024-04-24T22:35:05.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1283885056 bytes 2024-04-24T22:35:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1334627328 bytes 2024-04-24T22:35:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1388527616 bytes 2024-04-24T22:35:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1429481472 bytes 2024-04-24T22:35:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1430085632 bytes 2024-04-24T22:35:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1467636736 bytes 2024-04-24T22:35:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1500380160 bytes 2024-04-24T22:35:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1541529600 bytes 2024-04-24T22:35:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1670189056 bytes 2024-04-24T22:35:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1713104896 bytes 2024-04-24T22:35:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1765005312 bytes 2024-04-24T22:35:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1787126784 bytes 2024-04-24T22:35:06.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1811653632 bytes 2024-04-24T22:35:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 1874580480 bytes 2024-04-24T22:35:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 1900058624 bytes 2024-04-24T22:35:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:35:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:35:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:35:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:35:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-24T22:35:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:35:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:35:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:35:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:35:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:35:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:35:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T22:35:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:35:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:35:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:35:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:35:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:35:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:35:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T22:35:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T22:35:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:35:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+replaying 2024-04-24T22:35:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535440384.0,"last_snapshot_bytes":535820288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713998044,"remote_snapshot_timestamp":1713998044,"replay_state":"idle"} 2024-04-24T22:35:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: service: mirror.1 on smithi134 2024-04-24T22:35:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:34:53 2024-04-24T22:35:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T22:35:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T22:35:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T22:35:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T22:35:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:34:58' 2024-04-24T22:35:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T22:35:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: state: up+replaying 2024-04-24T22:35:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ description: .*' 2024-04-24T22:35:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535440384.0,"last_snapshot_bytes":535820288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713998044,"remote_snapshot_timestamp":1713998044,"replay_state":"idle"} 2024-04-24T22:35:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:35:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T22:35:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^up+' 2024-04-24T22:35:08.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T22:35:08.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T22:35:08.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: service: mirror.1 on smithi134 2024-04-24T22:35:08.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update 2024-04-24T22:35:08.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T22:35:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 22:34:53' 2024-04-24T22:35:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-24 22:34:53' 2024-04-24T22:35:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:35:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:35:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'prev_last_update=2024-04-24 22:34:53' 2024-04-24T22:35:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:35:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:35:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T22:35:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='OKOKOK142044313mirror.0smithi13417.2.7-915-g330067dbfalseOK42164316mirror.3smithi13417.2.7-915-g330067dbfalseOK42804283mirror.1smithi13417.2.7-915-g330067dbtrueOK44654468mirror.2smithi13417.2.7-915-g330067dbfalseOKtest829c41fd-bfa9-46e0-a6ae-748aa71d3214up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535440384.0,"last_snapshot_bytes":535820288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713998044,"remote_snapshot_timestamp":1713998044,"replay_state":"idle"}42804283mirror.1smithi1342024-04-24 22:34:53cluster2d9b73256-b8ac-4751-9b60-95a0234fa37cup+stoppedlocal image is primary2024-04-24 22:34:58' 2024-04-24T22:35:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update state 2024-04-24T22:35:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-24T22:35:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 22:34:53' 2024-04-24T22:35:08.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-24T22:35:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=up+replaying 2024-04-24T22:35:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T22:35:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep up+replaying 2024-04-24T22:35:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T22:35:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:35:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:35:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap7 2024-04-24T22:35:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap7 2024-04-24T22:35:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:35:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap7 2024-04-24T22:35:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:35:21.474 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9038981068034692) > weight (0.25) 2024-04-24T22:35:21.475 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.36212867387795167) > weight (0.25) 2024-04-24T22:35:21.475 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-04-24T22:35:21.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-04-24T22:35:21.475 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-04-24T22:35:21.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T22:35:21.475+0000 7f608da7e700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 52170) UID: 1000 2024-04-24T22:35:21.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:35:28.477 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-04-24T22:35:28.477 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:35:28.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:35:28.478 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-04-24T22:35:28.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-24T22:35:28.478 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-24T22:35:28.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-24T22:35:28.497 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2024-04-24T22:35:33.372 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-24T22:35:33.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-24T22:35:33.373 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-04-24T22:35:33.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:35:33.372+0000 7f53ceff2700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 48019) UID: 1000 2024-04-24T22:35:33.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:35:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T22:35:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:35:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T22:35:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:35:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:35:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:35:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:35:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:35:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T22:35:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:35:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:35:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:35:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:35:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T22:35:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:35:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:35:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:35:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:35:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:35:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:35:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:35:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:35:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:35:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:35:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:35:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T22:35:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:35:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:35:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:35:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:35:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:35:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:35:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:35:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:35:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:35:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:35:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:35:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:35:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:35:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:35:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:35:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:35:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:35:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:35:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:35:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:35:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:35:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:35:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:35:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:35:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:35:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:35:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:35:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:35:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:35:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:35:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:35:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:35:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:35:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T22:35:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:35:39.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:35:39.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:35:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:35:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T22:35:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:35:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:35:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:35:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:35:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:35:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:35:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:35:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:35:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:35:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:35:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:35:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:35:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:35:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:35:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:35:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:35:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:35:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:35:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:35:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:35:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:35:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:35:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:35:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:35:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:35:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:35:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:35:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:35:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:35:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:35:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:35:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:35:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:35:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T22:35:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:35:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:35:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:35:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:35:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:35:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:35:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:35:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:35:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-24T22:35:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:35:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:35:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:35:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:35:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-24T22:35:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:35:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:35:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-test.status 2024-04-24T22:35:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-test.status 2024-04-24T22:35:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:35:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:35:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-test.status 2024-04-24T22:35:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-test.status 2024-04-24T22:35:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:35:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:35:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-24T22:35:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:35:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-24T22:35:41.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 29 2024-04-24T22:35:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:35:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:35:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:35:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:35:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-24T22:35:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:35:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=29 2024-04-24T22:35:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:35:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:35:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:35:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:35:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:35:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:35:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:35:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:35:41.373 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-24T22:35:41.373 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:35:41.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:35:41.374 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-24T22:35:41.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-24T22:35:41.374 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-24T22:35:41.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-24T22:35:41.376 INFO:tasks.rbd_mirror_thrash:waiting for 114 secs before thrashing 2024-04-24T22:35:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:35:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:35:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:35:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:35:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:35:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:35:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 29 = '' 2024-04-24T22:35:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:35:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:35:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:35:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:35:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:35:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:35:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:35:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:35:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:35:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:35:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:35:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:35:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:35:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 29 = '' 2024-04-24T22:35:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:35:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:35:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:35:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:35:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:35:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:35:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:35:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:35:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:35:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:35:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:35:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:35:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:35:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 29 = '' 2024-04-24T22:35:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:35:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T22:35:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:35:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:35:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:35:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:35:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:35:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:35:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=29 2024-04-24T22:35:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 29 = 29 2024-04-24T22:35:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:35:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-04-24T22:35:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:35:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap7 2024-04-24T22:35:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:35:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:35:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:35:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-04-24T22:35:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'test': 2024-04-24T22:35:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T22:35:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:35:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 14 2024-04-24T22:35:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10bb1f6f9c8b 2024-04-24T22:35:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10bb1f6f9c8b 2024-04-24T22:35:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:35:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:35:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:35:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:35:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:35:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:34:58 2024 2024-04-24T22:35:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:35:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:35:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:35:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:35:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:35:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:35:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:35:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snap6-clean ']' 2024-04-24T22:35:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap6-clean 2024-04-24T22:35:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap6-clean 2024-04-24T22:35:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:35:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:35:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:35:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:35:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:35:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:35:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:35:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:35:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:35:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:35:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:35:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap7 2024-04-24T22:35:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap7 2024-04-24T22:35:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:35:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:35:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:35:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:35:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:35:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:35:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:35:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:35:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:35:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:35:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:35:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ clean_snap_name=snap7-clean 2024-04-24T22:35:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-04-24T22:35:48.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:35:48.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:48.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:48.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap7-clean 2024-04-24T22:35:48.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-04-24T22:35:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:35:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror test 2024-04-24T22:35:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:35:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:35:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=7 2024-04-24T22:35:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:35:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-24T22:35:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 38052864 bytes 2024-04-24T22:35:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 79954944 bytes 2024-04-24T22:35:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 133529600 bytes 2024-04-24T22:35:49.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 210285568 bytes 2024-04-24T22:35:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 238671872 bytes 2024-04-24T22:35:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 336088064 bytes 2024-04-24T22:35:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 467335168 bytes 2024-04-24T22:35:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 530748416 bytes 2024-04-24T22:35:50.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 569876480 bytes 2024-04-24T22:35:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 649079808 bytes 2024-04-24T22:35:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 685168640 bytes 2024-04-24T22:35:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 714370048 bytes 2024-04-24T22:35:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 753335296 bytes 2024-04-24T22:35:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 764062720 bytes 2024-04-24T22:35:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 804022272 bytes 2024-04-24T22:35:51.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 815929344 bytes 2024-04-24T22:35:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 829489152 bytes 2024-04-24T22:35:51.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 887670784 bytes 2024-04-24T22:35:51.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 953112576 bytes 2024-04-24T22:35:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 1002518528 bytes 2024-04-24T22:35:51.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 1073024000 bytes 2024-04-24T22:35:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 1107014656 bytes 2024-04-24T22:35:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1209318400 bytes 2024-04-24T22:35:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1247511552 bytes 2024-04-24T22:35:52.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1248107520 bytes 2024-04-24T22:35:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1268826112 bytes 2024-04-24T22:35:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1269440512 bytes 2024-04-24T22:35:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1296111616 bytes 2024-04-24T22:35:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1343314944 bytes 2024-04-24T22:35:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1379787776 bytes 2024-04-24T22:35:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1381569536 bytes 2024-04-24T22:35:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1450435584 bytes 2024-04-24T22:35:53.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1480083456 bytes 2024-04-24T22:35:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1480680448 bytes 2024-04-24T22:35:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1503266816 bytes 2024-04-24T22:35:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1551030272 bytes 2024-04-24T22:35:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1632687104 bytes 2024-04-24T22:35:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1658641408 bytes 2024-04-24T22:35:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1721678848 bytes 2024-04-24T22:35:53.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1769403392 bytes 2024-04-24T22:35:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1807167488 bytes 2024-04-24T22:35:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1821544448 bytes 2024-04-24T22:35:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1831032832 bytes 2024-04-24T22:35:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1841488896 bytes 2024-04-24T22:35:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1871176704 bytes 2024-04-24T22:35:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1921361920 bytes 2024-04-24T22:35:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1962940416 bytes 2024-04-24T22:35:54.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2001338368 bytes 2024-04-24T22:35:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2014393344 bytes 2024-04-24T22:35:55.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2058699776 bytes 2024-04-24T22:35:55.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2095439872 bytes 2024-04-24T22:35:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2132387840 bytes 2024-04-24T22:35:55.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2166069248 bytes 2024-04-24T22:35:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2199542784 bytes 2024-04-24T22:35:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2229385216 bytes 2024-04-24T22:35:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2276701184 bytes 2024-04-24T22:35:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2318057472 bytes 2024-04-24T22:35:55.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2350096384 bytes 2024-04-24T22:35:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2402549760 bytes 2024-04-24T22:35:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:35:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:35:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:35:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:35:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-24T22:35:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:35:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:35:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:35:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:35:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:35:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:35:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T22:35:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:35:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:35:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:35:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:35:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:35:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:35:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T22:35:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T22:35:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:35:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+replaying 2024-04-24T22:35:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: replaying, {"bytes_per_second":17362944.0,"bytes_per_snapshot":527690240.0,"last_snapshot_bytes":520888320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713998140,"remote_snapshot_timestamp":1713998140,"replay_state":"idle"} 2024-04-24T22:35:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: service: mirror.1 on smithi134 2024-04-24T22:35:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:35:53 2024-04-24T22:35:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T22:35:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T22:35:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T22:35:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T22:35:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:35:37' 2024-04-24T22:35:57.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T22:35:57.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: state: up+replaying 2024-04-24T22:35:57.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ description: .*' 2024-04-24T22:35:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: description: replaying, {"bytes_per_second":17362944.0,"bytes_per_snapshot":527690240.0,"last_snapshot_bytes":520888320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713998140,"remote_snapshot_timestamp":1713998140,"replay_state":"idle"} 2024-04-24T22:35:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:35:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T22:35:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^up+' 2024-04-24T22:35:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T22:35:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T22:35:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: service: mirror.1 on smithi134 2024-04-24T22:35:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update 2024-04-24T22:35:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T22:35:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 22:35:53' 2024-04-24T22:35:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-24 22:35:53' 2024-04-24T22:35:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:35:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:35:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'prev_last_update=2024-04-24 22:35:53' 2024-04-24T22:35:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:35:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:35:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T22:35:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='OKOKOK142044313mirror.0smithi13417.2.7-915-g330067dbfalseOK42164316mirror.3smithi13417.2.7-915-g330067dbfalseOK42804283mirror.1smithi13417.2.7-915-g330067dbtrueOK45384541mirror.2smithi13417.2.7-915-g330067dbfalseOKtest829c41fd-bfa9-46e0-a6ae-748aa71d3214up+replayingreplaying, {"bytes_per_second":17362944.0,"bytes_per_snapshot":527690240.0,"last_snapshot_bytes":520888320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713998140,"remote_snapshot_timestamp":1713998140,"replay_state":"idle"}42804283mirror.1smithi1342024-04-24 22:35:53cluster2d9b73256-b8ac-4751-9b60-95a0234fa37cup+stoppedlocal image is primary2024-04-24 22:35:37' 2024-04-24T22:35:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update state 2024-04-24T22:35:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-24T22:35:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 22:35:53' 2024-04-24T22:35:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-24T22:35:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=up+replaying 2024-04-24T22:35:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T22:35:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep up+replaying 2024-04-24T22:35:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T22:35:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:35:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:35:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap8 2024-04-24T22:35:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap8 2024-04-24T22:35:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:35:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:35:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:35:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap8 2024-04-24T22:35:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:36:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T22:36:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:36:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:36:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:36:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T22:36:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:36:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:36:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:36:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:36:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:36:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:36:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:36:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T22:36:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:36:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:36:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:36:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:36:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:36:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:36:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T22:36:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:36:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:36:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:36:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:36:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:36:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:36:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:36:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:36:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:36:28.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:36:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:36:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T22:36:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:36:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:36:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:36:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:36:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:36:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:36:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:36:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:36:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:36:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:36:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:36:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:36:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:36:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:36:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:36:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:36:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:36:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:36:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:36:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:36:28.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:36:28.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:36:28.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:36:28.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:36:28.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:36:28.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:36:28.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:36:28.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:36:28.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:36:28.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:36:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:36:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:36:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:36:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T22:36:28.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:36:28.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:36:28.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:36:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:36:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T22:36:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:36:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:36:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:36:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:36:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:36:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:36:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:36:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:36:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:36:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:36:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:36:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:36:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:36:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:36:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:36:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:36:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:36:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:36:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:36:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:36:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:36:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:36:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:36:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:36:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:36:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:36:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:36:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:36:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:36:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:36:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:36:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:36:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:36:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T22:36:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:36:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:36:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:36:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:36:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:36:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:36:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:36:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:36:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-24T22:36:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:36:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:36:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:36:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:36:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:36:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:36:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-24T22:36:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:36:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:36:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:36:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:36:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-test.status 2024-04-24T22:36:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-test.status 2024-04-24T22:36:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:36:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:36:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-test.status 2024-04-24T22:36:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-test.status 2024-04-24T22:36:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:36:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:36:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-24T22:36:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:36:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:36:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:36:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-24T22:36:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 33 2024-04-24T22:36:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:36:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:36:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:36:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:36:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:36:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-24T22:36:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:36:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:36:30.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=33 2024-04-24T22:36:30.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:36:30.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:36:30.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:36:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:36:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:36:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:36:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:36:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:36:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:36:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:36:30.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:36:30.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:36:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:36:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:36:30.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:36:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:36:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 33 = '' 2024-04-24T22:36:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:36:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:36:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:36:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:36:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:36:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:36:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:36:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:36:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:36:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:36:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:36:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:36:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:36:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:36:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:36:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 33 = '' 2024-04-24T22:36:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:36:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:36:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:36:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:36:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:36:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:36:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:36:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:36:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:36:31.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:36:31.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:36:31.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:36:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:36:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:36:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:36:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 33 = '' 2024-04-24T22:36:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:36:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T22:36:32.497 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-24T22:36:32.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-24T22:36:32.498 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-04-24T22:36:32.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T22:36:32.498+0000 7ff7da144700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 47930) UID: 1000 2024-04-24T22:36:32.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:36:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:36:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:36:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:36:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:36:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:36:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:36:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:36:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:36:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=33 2024-04-24T22:36:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 33 = 33 2024-04-24T22:36:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:36:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-04-24T22:36:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:36:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:36:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:36:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap8 2024-04-24T22:36:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:36:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:36:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:36:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-04-24T22:36:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'test': 2024-04-24T22:36:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T22:36:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:36:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 16 2024-04-24T22:36:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10bb1f6f9c8b 2024-04-24T22:36:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10bb1f6f9c8b 2024-04-24T22:36:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:36:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:36:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:36:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:36:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:36:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:34:58 2024 2024-04-24T22:36:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:36:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:36:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:36:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:36:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:36:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:36:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:36:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snap7-clean ']' 2024-04-24T22:36:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap7-clean 2024-04-24T22:36:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:36:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:36:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap7-clean 2024-04-24T22:36:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:36:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:36:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:36:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:36:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:36:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:36:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:36:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:36:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:36:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:36:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:36:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap8 2024-04-24T22:36:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:36:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:36:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap8 2024-04-24T22:36:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:36:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:36:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:36:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:36:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:36:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:36:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:36:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:36:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:36:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:36:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:36:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ clean_snap_name=snap8-clean 2024-04-24T22:36:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-04-24T22:36:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:36:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:36:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:36:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap8-clean 2024-04-24T22:36:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-04-24T22:36:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:36:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror test 2024-04-24T22:36:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:36:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:36:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:36:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:36:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=10 2024-04-24T22:36:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:36:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-24T22:36:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 44599296 bytes 2024-04-24T22:36:38.499 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-24T22:36:38.499 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:36:38.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:36:38.499 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-24T22:36:38.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-24T22:36:38.500 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-24T22:36:38.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-24T22:36:38.502 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2024-04-24T22:36:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 137674752 bytes 2024-04-24T22:36:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 184179712 bytes 2024-04-24T22:36:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 203342848 bytes 2024-04-24T22:36:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 249351168 bytes 2024-04-24T22:36:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 330484736 bytes 2024-04-24T22:36:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 360474624 bytes 2024-04-24T22:36:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 448925696 bytes 2024-04-24T22:36:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 477651968 bytes 2024-04-24T22:36:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 566728704 bytes 2024-04-24T22:36:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 657082368 bytes 2024-04-24T22:36:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 695109632 bytes 2024-04-24T22:36:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 710975488 bytes 2024-04-24T22:36:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 751561728 bytes 2024-04-24T22:36:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 814219264 bytes 2024-04-24T22:36:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 844213248 bytes 2024-04-24T22:36:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 981666816 bytes 2024-04-24T22:36:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 1060895744 bytes 2024-04-24T22:36:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 1127435264 bytes 2024-04-24T22:36:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 1130276864 bytes 2024-04-24T22:36:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 1130895360 bytes 2024-04-24T22:36:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 1224201216 bytes 2024-04-24T22:36:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1239135232 bytes 2024-04-24T22:36:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1333685248 bytes 2024-04-24T22:36:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1372276736 bytes 2024-04-24T22:36:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1416965120 bytes 2024-04-24T22:36:41.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1458704384 bytes 2024-04-24T22:36:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1493684224 bytes 2024-04-24T22:36:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1517339648 bytes 2024-04-24T22:36:42.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1544452096 bytes 2024-04-24T22:36:42.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1608494080 bytes 2024-04-24T22:36:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1637380096 bytes 2024-04-24T22:36:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1646287872 bytes 2024-04-24T22:36:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1670476800 bytes 2024-04-24T22:36:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1692460032 bytes 2024-04-24T22:36:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1755170816 bytes 2024-04-24T22:36:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1770736640 bytes 2024-04-24T22:36:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1789930496 bytes 2024-04-24T22:36:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1804733440 bytes 2024-04-24T22:36:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1858924544 bytes 2024-04-24T22:36:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1886669824 bytes 2024-04-24T22:36:43.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1918772224 bytes 2024-04-24T22:36:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1919398912 bytes 2024-04-24T22:36:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1973594112 bytes 2024-04-24T22:36:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 2020862976 bytes 2024-04-24T22:36:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 2051220480 bytes 2024-04-24T22:36:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 2088521728 bytes 2024-04-24T22:36:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2139260928 bytes 2024-04-24T22:36:44.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2195155968 bytes 2024-04-24T22:36:44.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2231404544 bytes 2024-04-24T22:36:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2271100928 bytes 2024-04-24T22:36:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2362227712 bytes 2024-04-24T22:36:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2478829568 bytes 2024-04-24T22:36:44.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2511175680 bytes 2024-04-24T22:36:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2571491328 bytes 2024-04-24T22:36:45.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2606342144 bytes 2024-04-24T22:36:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2648572928 bytes 2024-04-24T22:36:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2684510208 bytes 2024-04-24T22:36:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2752696320 bytes 2024-04-24T22:36:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2770260992 bytes 2024-04-24T22:36:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2841423872 bytes 2024-04-24T22:36:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2907446272 bytes 2024-04-24T22:36:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2975715328 bytes 2024-04-24T22:36:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 3019815936 bytes 2024-04-24T22:36:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 3046066176 bytes 2024-04-24T22:36:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 3113476096 bytes 2024-04-24T22:36:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 3135545344 bytes 2024-04-24T22:36:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 3192085504 bytes 2024-04-24T22:36:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 3231433728 bytes 2024-04-24T22:36:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 3254600704 bytes 2024-04-24T22:36:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 3321078784 bytes 2024-04-24T22:36:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 3353113600 bytes 2024-04-24T22:36:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 3376027648 bytes 2024-04-24T22:36:47.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 3432413184 bytes 2024-04-24T22:36:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 3433003008 bytes 2024-04-24T22:36:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 3480382464 bytes 2024-04-24T22:36:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 3505900544 bytes 2024-04-24T22:36:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 3586318336 bytes 2024-04-24T22:36:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7900 IOs, 3628699648 bytes 2024-04-24T22:36:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-24T22:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:36:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:36:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T22:36:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:36:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:36:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:36:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:36:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:36:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:36:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:36:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:36:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T22:36:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T22:36:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:36:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+replaying 2024-04-24T22:36:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":527690240.0,"last_snapshot_bytes":520888320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713998140,"remote_snapshot_timestamp":1713998140,"replay_state":"idle"} 2024-04-24T22:36:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: service: mirror.1 on smithi134 2024-04-24T22:36:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:36:23 2024-04-24T22:36:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T22:36:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T22:36:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T22:36:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T22:36:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:36:36' 2024-04-24T22:36:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T22:36:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: state: up+replaying 2024-04-24T22:36:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ description: .*' 2024-04-24T22:36:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":527690240.0,"last_snapshot_bytes":520888320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713998140,"remote_snapshot_timestamp":1713998140,"replay_state":"idle"} 2024-04-24T22:36:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:36:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T22:36:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^up+' 2024-04-24T22:36:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T22:36:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T22:36:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: service: mirror.1 on smithi134 2024-04-24T22:36:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update 2024-04-24T22:36:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T22:36:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 22:36:23' 2024-04-24T22:36:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-24 22:36:23' 2024-04-24T22:36:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:36:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:36:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:36:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:36:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'prev_last_update=2024-04-24 22:36:23' 2024-04-24T22:36:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:36:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:36:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T22:36:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='OKOKOK142164316mirror.3smithi13417.2.7-915-g330067dbfalseOK42804283mirror.1smithi13417.2.7-915-g330067dbtrueOK45384541mirror.2smithi13417.2.7-915-g330067dbfalseOK45994602mirror.0smithi13417.2.7-915-g330067dbfalseOKtest829c41fd-bfa9-46e0-a6ae-748aa71d3214up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":527690240.0,"last_snapshot_bytes":520888320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713998140,"remote_snapshot_timestamp":1713998140,"replay_state":"idle"}42804283mirror.1smithi1342024-04-24 22:36:23cluster2d9b73256-b8ac-4751-9b60-95a0234fa37cup+stoppedlocal image is primary2024-04-24 22:36:36' 2024-04-24T22:36:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update state 2024-04-24T22:36:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-24T22:36:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 22:36:23' 2024-04-24T22:36:49.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-24T22:36:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=up+replaying 2024-04-24T22:36:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T22:36:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep up+replaying 2024-04-24T22:36:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T22:36:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:36:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:36:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap9 2024-04-24T22:36:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap9 2024-04-24T22:36:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:36:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:36:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:36:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap9 2024-04-24T22:36:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:37:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T22:37:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:37:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:37:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:37:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T22:37:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:37:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:37:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:37:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:37:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:37:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:37:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:37:20.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T22:37:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:37:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:37:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:37:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:37:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:37:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:37:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T22:37:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:37:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:37:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:37:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:37:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:37:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:37:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:37:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:37:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:37:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:37:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:37:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T22:37:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:37:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:37:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:37:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:37:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:37:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:37:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:37:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:37:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:37:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:37:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:37:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:37:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:37:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:37:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:37:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:37:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:37:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:37:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:37:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:37:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:37:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:37:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:37:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:37:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:37:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:37:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:37:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:37:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:37:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:37:20.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:37:20.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:37:20.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:37:20.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T22:37:20.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:37:20.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:37:20.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:37:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:37:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T22:37:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:37:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:37:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:37:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:37:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:37:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:37:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:37:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:37:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:37:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:37:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:37:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:37:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:37:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:37:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:37:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:37:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:37:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:37:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:37:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:37:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:37:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:37:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:37:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:37:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:37:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:37:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:37:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:37:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:37:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:37:20.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:37:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:37:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:37:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T22:37:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:37:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:37:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:37:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:37:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:37:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:37:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:37:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:37:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-24T22:37:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:37:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:37:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:37:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:37:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:37:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:37:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-24T22:37:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:37:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:37:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:37:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:37:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-test.status 2024-04-24T22:37:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-test.status 2024-04-24T22:37:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:37:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:37:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-test.status 2024-04-24T22:37:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-test.status 2024-04-24T22:37:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:37:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:37:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-24T22:37:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:37:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:37:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:37:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-24T22:37:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 37 2024-04-24T22:37:22.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:37:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:37:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:37:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:37:22.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:37:22.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-24T22:37:22.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:37:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:37:22.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=37 2024-04-24T22:37:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:37:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:37:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:37:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:37:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:37:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:37:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:37:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:37:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:37:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:37:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:37:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:37:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:37:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:37:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:37:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:37:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 37 = '' 2024-04-24T22:37:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:37:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:37:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:37:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:37:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:37:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:37:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:37:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:37:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:37:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:37:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:37:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:37:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:37:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:37:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:37:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 37 = '' 2024-04-24T22:37:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:37:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:37:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:37:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:37:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:37:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:37:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:37:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:37:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:37:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:37:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:37:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:37:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:37:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:37:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:37:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 37 = '' 2024-04-24T22:37:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:37:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T22:37:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:37:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:37:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:37:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:37:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:37:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:37:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:37:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:37:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=37 2024-04-24T22:37:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 37 = 37 2024-04-24T22:37:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:37:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-04-24T22:37:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:37:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:37:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:37:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap9 2024-04-24T22:37:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:37:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:37:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:37:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-04-24T22:37:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'test': 2024-04-24T22:37:26.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T22:37:26.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:37:26.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 18 2024-04-24T22:37:26.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10bb1f6f9c8b 2024-04-24T22:37:26.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10bb1f6f9c8b 2024-04-24T22:37:26.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:37:26.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:37:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:37:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:37:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:37:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:36:34 2024 2024-04-24T22:37:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:37:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:37:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:37:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:37:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:37:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:37:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:37:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snap8-clean ']' 2024-04-24T22:37:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap8-clean 2024-04-24T22:37:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:37:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:37:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap8-clean 2024-04-24T22:37:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:37:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:37:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:37:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:37:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:37:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:37:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:37:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:37:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:37:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:37:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:37:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap9 2024-04-24T22:37:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:37:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:37:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap9 2024-04-24T22:37:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:37:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:37:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:37:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:37:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:37:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:37:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:37:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:37:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:37:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:37:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:37:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ clean_snap_name=snap9-clean 2024-04-24T22:37:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-04-24T22:37:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:37:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:37:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:37:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap9-clean 2024-04-24T22:37:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-04-24T22:37:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:37:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror test 2024-04-24T22:37:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:37:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:37:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:37:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:37:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=14 2024-04-24T22:37:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:37:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-04-24T22:37:31.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 60625920 bytes 2024-04-24T22:37:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 76519424 bytes 2024-04-24T22:37:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 111400960 bytes 2024-04-24T22:37:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 159138816 bytes 2024-04-24T22:37:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 195568640 bytes 2024-04-24T22:37:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 218303488 bytes 2024-04-24T22:37:32.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 264236032 bytes 2024-04-24T22:37:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 286507008 bytes 2024-04-24T22:37:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 305038336 bytes 2024-04-24T22:37:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 331603968 bytes 2024-04-24T22:37:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 395563008 bytes 2024-04-24T22:37:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 484238336 bytes 2024-04-24T22:37:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 564781056 bytes 2024-04-24T22:37:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 623590400 bytes 2024-04-24T22:37:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 624330752 bytes 2024-04-24T22:37:33.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 643741696 bytes 2024-04-24T22:37:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 666725376 bytes 2024-04-24T22:37:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 729930752 bytes 2024-04-24T22:37:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 814564352 bytes 2024-04-24T22:37:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 895659008 bytes 2024-04-24T22:37:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 956344320 bytes 2024-04-24T22:37:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 1022002176 bytes 2024-04-24T22:37:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1050328064 bytes 2024-04-24T22:37:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1091154944 bytes 2024-04-24T22:37:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1113918464 bytes 2024-04-24T22:37:34.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1190591488 bytes 2024-04-24T22:37:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1255300096 bytes 2024-04-24T22:37:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1342291968 bytes 2024-04-24T22:37:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1396961280 bytes 2024-04-24T22:37:35.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1435432960 bytes 2024-04-24T22:37:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1536094208 bytes 2024-04-24T22:37:35.377 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3271337152110463) > weight (0.25) 2024-04-24T22:37:35.377 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9503221660056584) > weight (0.25) 2024-04-24T22:37:35.377 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-04-24T22:37:35.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-04-24T22:37:35.377 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-04-24T22:37:35.378 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T22:37:35.377+0000 7f27adbf2700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 48156) UID: 1000 2024-04-24T22:37:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1586489344 bytes 2024-04-24T22:37:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1634348032 bytes 2024-04-24T22:37:35.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:37:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1643782144 bytes 2024-04-24T22:37:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1674604544 bytes 2024-04-24T22:37:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1745691648 bytes 2024-04-24T22:37:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1822820352 bytes 2024-04-24T22:37:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1895236608 bytes 2024-04-24T22:37:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1941941248 bytes 2024-04-24T22:37:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1973456896 bytes 2024-04-24T22:37:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 2009657344 bytes 2024-04-24T22:37:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 2050139136 bytes 2024-04-24T22:37:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 2050718720 bytes 2024-04-24T22:37:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 2104305664 bytes 2024-04-24T22:37:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 2176240640 bytes 2024-04-24T22:37:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 2219938816 bytes 2024-04-24T22:37:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 2235522048 bytes 2024-04-24T22:37:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2270836736 bytes 2024-04-24T22:37:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2354135040 bytes 2024-04-24T22:37:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2416314368 bytes 2024-04-24T22:37:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2445620224 bytes 2024-04-24T22:37:37.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2489411584 bytes 2024-04-24T22:37:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2522662912 bytes 2024-04-24T22:37:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2589182976 bytes 2024-04-24T22:37:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2649578496 bytes 2024-04-24T22:37:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2728685568 bytes 2024-04-24T22:37:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2744231936 bytes 2024-04-24T22:37:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2779555840 bytes 2024-04-24T22:37:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2786634752 bytes 2024-04-24T22:37:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2869701632 bytes 2024-04-24T22:37:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2874901504 bytes 2024-04-24T22:37:39.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2939431936 bytes 2024-04-24T22:37:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 3017039872 bytes 2024-04-24T22:37:39.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 3091577856 bytes 2024-04-24T22:37:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 3148867584 bytes 2024-04-24T22:37:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 3226180608 bytes 2024-04-24T22:37:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 3268006912 bytes 2024-04-24T22:37:39.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 3297999872 bytes 2024-04-24T22:37:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 3309687808 bytes 2024-04-24T22:37:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 3363594240 bytes 2024-04-24T22:37:40.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 3402830848 bytes 2024-04-24T22:37:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 3448399872 bytes 2024-04-24T22:37:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 3473641472 bytes 2024-04-24T22:37:40.503 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5387707092580364) > weight (0.25) 2024-04-24T22:37:40.503 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3952551007529155) > weight (0.25) 2024-04-24T22:37:40.503 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7998901195443782) > weight (0.25) 2024-04-24T22:37:40.503 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5328558282549191) > weight (0.25) 2024-04-24T22:37:40.503 INFO:tasks.rbd_mirror_thrash:waiting for 97 secs before thrashing 2024-04-24T22:37:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 3494135808 bytes 2024-04-24T22:37:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 3549650944 bytes 2024-04-24T22:37:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 3572015104 bytes 2024-04-24T22:37:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 3618727936 bytes 2024-04-24T22:37:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 3651031040 bytes 2024-04-24T22:37:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7900 IOs, 3651873792 bytes 2024-04-24T22:37:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8000 IOs, 3725964288 bytes 2024-04-24T22:37:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8100 IOs, 3743475712 bytes 2024-04-24T22:37:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8200 IOs, 3784235008 bytes 2024-04-24T22:37:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8300 IOs, 3883392000 bytes 2024-04-24T22:37:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8400 IOs, 3893487616 bytes 2024-04-24T22:37:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8500 IOs, 3918685184 bytes 2024-04-24T22:37:41.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8600 IOs, 3960992768 bytes 2024-04-24T22:37:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8700 IOs, 3979874304 bytes 2024-04-24T22:37:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8800 IOs, 4047630336 bytes 2024-04-24T22:37:42.378 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-04-24T22:37:42.378 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:37:42.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:37:42.379 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-04-24T22:37:42.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-04-24T22:37:42.379 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-04-24T22:37:42.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-04-24T22:37:42.382 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2024-04-24T22:37:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8900 IOs, 4066583552 bytes 2024-04-24T22:37:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9000 IOs, 4126368768 bytes 2024-04-24T22:37:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9100 IOs, 4162860032 bytes 2024-04-24T22:37:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9200 IOs, 4221945856 bytes 2024-04-24T22:37:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9300 IOs, 4243550208 bytes 2024-04-24T22:37:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9400 IOs, 4303845376 bytes 2024-04-24T22:37:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9500 IOs, 4342941696 bytes 2024-04-24T22:37:43.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9600 IOs, 4380642304 bytes 2024-04-24T22:37:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9700 IOs, 4389451776 bytes 2024-04-24T22:37:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9800 IOs, 4404740096 bytes 2024-04-24T22:37:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9900 IOs, 4467922944 bytes 2024-04-24T22:37:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10000 IOs, 4564448256 bytes 2024-04-24T22:37:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10100 IOs, 4565251072 bytes 2024-04-24T22:37:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10200 IOs, 4635196416 bytes 2024-04-24T22:37:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10300 IOs, 4705131520 bytes 2024-04-24T22:37:44.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10400 IOs, 4741104640 bytes 2024-04-24T22:37:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10500 IOs, 4763725824 bytes 2024-04-24T22:37:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10600 IOs, 4836272128 bytes 2024-04-24T22:37:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10700 IOs, 4836852736 bytes 2024-04-24T22:37:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10800 IOs, 4883913728 bytes 2024-04-24T22:37:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10900 IOs, 4952707072 bytes 2024-04-24T22:37:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:37:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:37:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:37:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:37:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-04-24T22:37:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:37:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:37:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:37:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:37:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:37:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:37:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:37:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:37:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-04-24T22:37:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:37:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:37:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:37:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:37:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local description_pattern= 2024-04-24T22:37:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local service_pattern= 2024-04-24T22:37:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:37:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:37:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-04-24T22:37:46.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='test: 2024-04-24T22:37:46.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: global_id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:37:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+replaying 2024-04-24T22:37:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":527269376.0,"last_snapshot_bytes":533650432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713998189,"remote_snapshot_timestamp":1713998189,"replay_state":"idle"} 2024-04-24T22:37:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: service: mirror.1 on smithi134 2024-04-24T22:37:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:37:23 2024-04-24T22:37:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: peer_sites: 2024-04-24T22:37:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: name: cluster2 2024-04-24T22:37:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: state: up+stopped 2024-04-24T22:37:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: description: local image is primary 2024-04-24T22:37:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: last_update: 2024-04-24 22:37:36' 2024-04-24T22:37:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ state: .*up+replaying' 2024-04-24T22:37:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: state: up+replaying 2024-04-24T22:37:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^ description: .*' 2024-04-24T22:37:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":527269376.0,"last_snapshot_bytes":533650432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713998189,"remote_snapshot_timestamp":1713998189,"replay_state":"idle"} 2024-04-24T22:37:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:37:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T22:37:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep '^up+' 2024-04-24T22:37:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T22:37:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T22:37:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: service: mirror.1 on smithi134 2024-04-24T22:37:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update 2024-04-24T22:37:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T22:37:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 22:37:23' 2024-04-24T22:37:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-04-24 22:37:23' 2024-04-24T22:37:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:37:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:37:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:37:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state_pattern=up+replaying 2024-04-24T22:37:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local 'prev_last_update=2024-04-24 22:37:23' 2024-04-24T22:37:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status 2024-04-24T22:37:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ CEPH_ARGS= 2024-04-24T22:37:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T22:37:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status='OKOKOK142164316mirror.3smithi13417.2.7-915-g330067dbfalseOK42804283mirror.1smithi13417.2.7-915-g330067dbtrueOK45384541mirror.2smithi13417.2.7-915-g330067dbfalseOK45994602mirror.0smithi13417.2.7-915-g330067dbfalseOKtest829c41fd-bfa9-46e0-a6ae-748aa71d3214up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":527269376.0,"last_snapshot_bytes":533650432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1713998189,"remote_snapshot_timestamp":1713998189,"replay_state":"idle"}42804283mirror.1smithi1342024-04-24 22:37:23cluster2d9b73256-b8ac-4751-9b60-95a0234fa37cup+stoppedlocal image is primary2024-04-24 22:37:36' 2024-04-24T22:37:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_update state 2024-04-24T22:37:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-04-24T22:37:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ last_update='2024-04-24 22:37:23' 2024-04-24T22:37:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-04-24T22:37:46.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=up+replaying 2024-04-24T22:37:46.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo up+replaying 2024-04-24T22:37:46.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep up+replaying 2024-04-24T22:37:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:up+replaying 2024-04-24T22:37:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:37:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:37:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap10 2024-04-24T22:37:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap10 2024-04-24T22:37:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:37:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:37:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:37:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap10 2024-04-24T22:37:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:38:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T22:38:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:38:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T22:38:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:38:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:38:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:38:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:38:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:38:16.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T22:38:16.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:38:16.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:16.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:16.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:38:16.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:38:16.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:38:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T22:38:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:38:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:38:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:38:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:38:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:38:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:38:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:38:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:38:16.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:38:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:38:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:38:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T22:38:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:38:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:38:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:38:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:38:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:38:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:38:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:38:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:38:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:38:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:38:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:38:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:38:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:38:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:38:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:38:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:38:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:38:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:38:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:38:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:38:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:38:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:38:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:38:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:38:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:38:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:38:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:38:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:38:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:38:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:38:16.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:38:16.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:38:16.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:38:16.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T22:38:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:38:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:38:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:38:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:38:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T22:38:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:38:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:38:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:38:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:38:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:38:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:38:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:38:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:38:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:38:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:38:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:38:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:38:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:38:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:38:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:38:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:38:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:38:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:38:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:38:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:38:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:38:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:38:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:38:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:38:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:38:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:38:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:38:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:38:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:38:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:38:16.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:38:16.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:38:16.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:38:16.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T22:38:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:38:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:38:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:38:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:38:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:38:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:38:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:38:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-24T22:38:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:38:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:38:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:38:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:38:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-24T22:38:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:38:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:38:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-test.status 2024-04-24T22:38:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-test.status 2024-04-24T22:38:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:38:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:38:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-test.status 2024-04-24T22:38:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-test.status 2024-04-24T22:38:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:38:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:38:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-24T22:38:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:38:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-24T22:38:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 41 2024-04-24T22:38:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:38:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:38:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:38:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-24T22:38:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:38:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:38:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=41 2024-04-24T22:38:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:38:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:38:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:38:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:38:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:38:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:38:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:38:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:38:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:38:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:38:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:38:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:38:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:38:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:38:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 41 = '' 2024-04-24T22:38:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:38:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:38:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:38:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:38:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:38:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:38:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:38:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:38:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:38:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:38:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:38:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:38:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:38:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 41 = '' 2024-04-24T22:38:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:38:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:38:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:38:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:38:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:38:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:38:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:38:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:38:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:38:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:38:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:38:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:38:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:38:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 41 = '' 2024-04-24T22:38:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:38:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T22:38:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:38:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:38:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:38:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:38:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:38:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:38:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=41 2024-04-24T22:38:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 41 = 41 2024-04-24T22:38:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-04-24T22:38:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:38:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap10 2024-04-24T22:38:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:38:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:38:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:38:22.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-04-24T22:38:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'test': 2024-04-24T22:38:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T22:38:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:38:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 20 2024-04-24T22:38:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10bb1f6f9c8b 2024-04-24T22:38:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10bb1f6f9c8b 2024-04-24T22:38:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:38:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:38:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:38:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:38:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:38:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:36:34 2024 2024-04-24T22:38:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:38:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:38:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:38:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:38:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:38:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:38:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snap9-clean ']' 2024-04-24T22:38:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap9-clean 2024-04-24T22:38:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap9-clean 2024-04-24T22:38:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:38:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:23.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:23.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap10 2024-04-24T22:38:24.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:24.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:24.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap10 2024-04-24T22:38:24.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:38:24.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:24.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:24.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:24.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:24.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ clean_snap_name=snap10-clean 2024-04-24T22:38:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-04-24T22:38:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:38:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap10-clean 2024-04-24T22:38:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-04-24T22:38:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-04-24T22:38:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:38:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-04-24T22:38:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:38:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:38:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:38:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:38:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:38:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror test started 2024-04-24T22:38:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:38:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:38:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:38:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:38:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-04-24T22:38:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:38:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:38:27.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:38:27.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:38:27.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:38:27.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:38:27.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:38:27.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:38:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:38:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:38:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:38:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-04-24T22:38:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:38:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:38:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:38:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:38:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:38:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:38:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:38:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:38:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:38:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:38:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:38:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:38:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:38:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:38:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:38:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:38:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:38:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:38:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:38:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:38:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:38:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:38:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:38:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:38:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:38:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:38:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:38:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:38:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:38:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:38:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:38:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:38:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:38:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-04-24T22:38:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-04-24T22:38:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:38:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:38:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:38:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-04-24T22:38:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:38:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:38:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:38:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:38:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:38:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:38:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:38:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:38:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:38:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:38:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:38:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:38:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:38:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:38:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:38:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:38:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:38:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:38:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:38:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:38:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:38:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:38:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:38:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:38:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:38:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:38:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:38:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:38:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:38:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:38:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:38:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:38:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:38:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-04-24T22:38:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:38:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:38:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:38:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:38:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:38:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:38:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:38:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-04-24T22:38:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:38:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:38:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:38:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:38:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-04-24T22:38:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:38:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:38:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:27.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-test.status 2024-04-24T22:38:27.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-test.status 2024-04-24T22:38:27.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:38:27.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:38:27.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-test.status 2024-04-24T22:38:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-test.status 2024-04-24T22:38:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:38:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:38:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-04-24T22:38:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:38:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-04-24T22:38:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 44 2024-04-24T22:38:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:38:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:38:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.status 2024-04-24T22:38:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-04-24T22:38:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:38:29.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:38:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=44 2024-04-24T22:38:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:38:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:38:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:38:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:38:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:38:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.status 2024-04-24T22:38:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-04-24T22:38:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:38:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:38:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=44 2024-04-24T22:38:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 44 = 44 2024-04-24T22:38:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-04-24T22:38:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:38:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap10-clean 2024-04-24T22:38:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:38:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:38:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:38:30.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-04-24T22:38:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'test': 2024-04-24T22:38:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T22:38:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:38:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 21 2024-04-24T22:38:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10bb1f6f9c8b 2024-04-24T22:38:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10bb1f6f9c8b 2024-04-24T22:38:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:38:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:38:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:38:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:38:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:38:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:38:23 2024 2024-04-24T22:38:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:29:55 2024 2024-04-24T22:38:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:38:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:38:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:38:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 829c41fd-bfa9-46e0-a6ae-748aa71d3214 2024-04-24T22:38:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:38:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 10 2024-04-24T22:38:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:38:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap1 2024-04-24T22:38:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap1 2024-04-24T22:38:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap1 2024-04-24T22:38:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:38:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:31.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:32.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:32.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap1-clean 2024-04-24T22:38:32.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap1-clean 2024-04-24T22:38:32.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap1-clean 2024-04-24T22:38:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:38:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:38:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap2 2024-04-24T22:38:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap2 2024-04-24T22:38:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap2 2024-04-24T22:38:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:38:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap2-clean 2024-04-24T22:38:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap2-clean 2024-04-24T22:38:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap2-clean 2024-04-24T22:38:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:38:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:36.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:38:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap3 2024-04-24T22:38:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap3 2024-04-24T22:38:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap3 2024-04-24T22:38:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:38:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:38.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap3-clean 2024-04-24T22:38:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap3-clean 2024-04-24T22:38:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap3-clean 2024-04-24T22:38:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:38:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:38:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap4 2024-04-24T22:38:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap4 2024-04-24T22:38:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap4 2024-04-24T22:38:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:38:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:40.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:40.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:40.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap4-clean 2024-04-24T22:38:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap4-clean 2024-04-24T22:38:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap4-clean 2024-04-24T22:38:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:38:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:38:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap5 2024-04-24T22:38:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap5 2024-04-24T22:38:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap5 2024-04-24T22:38:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:38:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap5-clean 2024-04-24T22:38:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap5-clean 2024-04-24T22:38:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap5-clean 2024-04-24T22:38:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:38:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:38:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap6 2024-04-24T22:38:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap6 2024-04-24T22:38:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap6 2024-04-24T22:38:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:38:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap6-clean 2024-04-24T22:38:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap6-clean 2024-04-24T22:38:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap6-clean 2024-04-24T22:38:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:38:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:49.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:38:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap7 2024-04-24T22:38:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap7 2024-04-24T22:38:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap7 2024-04-24T22:38:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:38:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:51.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap7-clean 2024-04-24T22:38:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap7-clean 2024-04-24T22:38:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap7-clean 2024-04-24T22:38:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:38:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:53.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:53.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:38:53.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap8 2024-04-24T22:38:53.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap8 2024-04-24T22:38:53.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap8 2024-04-24T22:38:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:38:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:54.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap8-clean 2024-04-24T22:38:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap8-clean 2024-04-24T22:38:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap8-clean 2024-04-24T22:38:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:38:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:56.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:56.383 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7094328373177876) > weight (0.25) 2024-04-24T22:38:56.383 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5551215856586716) > weight (0.25) 2024-04-24T22:38:56.383 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-04-24T22:38:56.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-04-24T22:38:56.384 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-04-24T22:38:56.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T22:38:56.383+0000 7fdbdd798700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 54726) UID: 1000 2024-04-24T22:38:56.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:38:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap9 2024-04-24T22:38:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap9 2024-04-24T22:38:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap9 2024-04-24T22:38:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:38:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:56.588 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:38:57.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap9-clean 2024-04-24T22:38:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap9-clean 2024-04-24T22:38:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap9-clean 2024-04-24T22:38:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:38:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:58.385 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-04-24T22:38:58.385 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:38:58.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:38:58.385 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-04-24T22:38:58.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-04-24T22:38:58.386 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-04-24T22:38:58.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-04-24T22:38:58.388 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2024-04-24T22:38:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:59.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:38:59.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:38:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:38:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap10 2024-04-24T22:38:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap10 2024-04-24T22:38:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:38:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:38:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap10 2024-04-24T22:38:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:38:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:38:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:38:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:39:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:39:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:39:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:39:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:39:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:39:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:39:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap10-clean 2024-04-24T22:39:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror test snap10-clean 2024-04-24T22:39:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap10-clean 2024-04-24T22:39:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:39:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:39:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:39:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:39:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export 2024-04-24T22:39:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:39:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:39:02.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-04-24T22:39:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:39:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-test.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-test.export 2024-04-24T22:39:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:39:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 10 2024-04-24T22:39:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:39:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap1 2024-04-24T22:39:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-04-24T22:39:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:03.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap1 2024-04-24T22:39:03.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-04-24T22:39:04.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:39:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap1-clean 2024-04-24T22:39:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-04-24T22:39:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap1-clean 2024-04-24T22:39:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-04-24T22:39:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:39:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:39:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap2 2024-04-24T22:39:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-04-24T22:39:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap2 2024-04-24T22:39:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-04-24T22:39:06.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:39:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap2-clean 2024-04-24T22:39:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-04-24T22:39:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap2-clean 2024-04-24T22:39:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-04-24T22:39:07.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:39:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:39:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap3 2024-04-24T22:39:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-04-24T22:39:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap3 2024-04-24T22:39:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-04-24T22:39:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:39:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap3-clean 2024-04-24T22:39:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-04-24T22:39:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap3-clean 2024-04-24T22:39:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-04-24T22:39:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:39:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:39:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap4 2024-04-24T22:39:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-04-24T22:39:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap4 2024-04-24T22:39:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-04-24T22:39:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:39:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap4-clean 2024-04-24T22:39:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-04-24T22:39:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:10.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:10.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:10.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap4-clean 2024-04-24T22:39:10.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-04-24T22:39:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:39:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:39:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap5 2024-04-24T22:39:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-04-24T22:39:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap5 2024-04-24T22:39:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-04-24T22:39:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:39:12.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap5-clean 2024-04-24T22:39:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-04-24T22:39:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap5-clean 2024-04-24T22:39:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-04-24T22:39:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:39:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:39:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap6 2024-04-24T22:39:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-04-24T22:39:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap6 2024-04-24T22:39:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-04-24T22:39:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:39:14.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap6-clean 2024-04-24T22:39:14.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-04-24T22:39:14.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:14.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:14.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:14.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap6-clean 2024-04-24T22:39:14.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-04-24T22:39:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:39:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:39:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap7 2024-04-24T22:39:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-04-24T22:39:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap7 2024-04-24T22:39:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-04-24T22:39:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:39:16.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap7-clean 2024-04-24T22:39:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-04-24T22:39:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap7-clean 2024-04-24T22:39:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-04-24T22:39:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:39:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:39:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap8 2024-04-24T22:39:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-04-24T22:39:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap8 2024-04-24T22:39:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-04-24T22:39:17.504 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5816969563405183) > weight (0.25) 2024-04-24T22:39:17.504 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-04-24T22:39:17.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-04-24T22:39:17.505 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-04-24T22:39:17.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T22:39:17.504+0000 7f32a2da7700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 49458) UID: 1000 2024-04-24T22:39:17.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:39:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:39:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap8-clean 2024-04-24T22:39:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-04-24T22:39:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap8-clean 2024-04-24T22:39:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-04-24T22:39:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:39:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:39:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap9 2024-04-24T22:39:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-04-24T22:39:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap9 2024-04-24T22:39:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-04-24T22:39:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:39:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap9-clean 2024-04-24T22:39:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-04-24T22:39:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap9-clean 2024-04-24T22:39:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-04-24T22:39:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:39:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:39:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap10 2024-04-24T22:39:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-04-24T22:39:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap10 2024-04-24T22:39:21.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-04-24T22:39:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:39:22.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap10-clean 2024-04-24T22:39:22.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-04-24T22:39:22.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:22.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:22.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:22.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap10-clean 2024-04-24T22:39:22.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-04-24T22:39:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:39:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror test 2024-04-24T22:39:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:39:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:39:23.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror test 2024-04-24T22:39:23.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:23.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:23.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-04-24T22:39:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-04-24T22:39:23.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:39:23.252+0000 7fecf939d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2ad6a340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:39:23.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:39:23.252+0000 7fecffbaa700 -1 rbd::mirror::ImageReplayer: 0x557d27914c80 [3/829c41fd-bfa9-46e0-a6ae-748aa71d3214] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:39:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-24T22:39:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:39:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-04-24T22:39:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:39:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:39:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:39:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:39:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:39:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror test 2024-04-24T22:39:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:39:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:39:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T22:39:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-04-24T22:39:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:39:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image test: (2) No such file or directory 2024-04-24T22:39:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:39:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:39:25.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:39:25.505 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-04-24T22:39:25.505 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:39:25.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:39:25.506 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-04-24T22:39:25.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-24T22:39:25.506 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-24T22:39:25.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-24T22:39:25.508 INFO:tasks.rbd_mirror_thrash:waiting for 103 secs before thrashing 2024-04-24T22:39:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror test deleted '' 2024-04-24T22:39:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:39:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=test 2024-04-24T22:39:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:39:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:39:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:39:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:39:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror test 2024-04-24T22:39:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:39:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:39:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=test 2024-04-24T22:39:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-04-24T22:39:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:39:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image test: (2) No such file or directory 2024-04-24T22:39:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:39:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:39:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:39:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:39:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ testlog 'TEST: create many images' 2024-04-24T22:39:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ tee -a /tmp/tmp.DCvHjwkiaM/rbd-mirror.test.log 2024-04-24T22:39:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ date '+%F %T' 2024-04-24T22:39:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo 2024-04-24 22:39:25 TEST: create many images 2024-04-24T22:39:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:2024-04-24 22:39:25 TEST: create many images 2024-04-24T22:39:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ snap_name=snap 2024-04-24T22:39:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 50 2024-04-24T22:39:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:39:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_1 2024-04-24T22:39:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2024-04-24T22:39:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:39:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:39:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:39:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_1 128M 2024-04-24T22:39:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:25.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:25.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:25.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:25.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:39:25.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:25.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:39:25.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:39:25.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:39:25.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:25.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2024-04-24T22:39:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:39:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2024-04-24T22:39:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:39:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:39:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-04-24T22:39:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:39:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-04-24T22:39:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_1': 2024-04-24T22:39:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:39:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:39:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:39:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 126da9d0dfca 2024-04-24T22:39:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.126da9d0dfca 2024-04-24T22:39:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:39:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:39:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:39:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:39:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:39:25 2024 2024-04-24T22:39:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:39:25 2024 2024-04-24T22:39:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:39:25 2024 2024-04-24T22:39:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:39:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:39:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: a0aa7e18-e06a-4e3c-b4d7-6bdccb4d22a8 2024-04-24T22:39:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:39:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:39:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_1 2024-04-24T22:39:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:39:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:39:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=6 2024-04-24T22:39:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:39:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:39:26.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 41869312 bytes 2024-04-24T22:39:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 102623232 bytes 2024-04-24T22:39:26.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 165256192 bytes 2024-04-24T22:39:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 226995200 bytes 2024-04-24T22:39:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 263399424 bytes 2024-04-24T22:39:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 297371648 bytes 2024-04-24T22:39:27.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 323809280 bytes 2024-04-24T22:39:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 370520064 bytes 2024-04-24T22:39:27.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 418169856 bytes 2024-04-24T22:39:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 450805760 bytes 2024-04-24T22:39:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 509549568 bytes 2024-04-24T22:39:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 551104512 bytes 2024-04-24T22:39:27.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 618180608 bytes 2024-04-24T22:39:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 636591104 bytes 2024-04-24T22:39:28.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 661270528 bytes 2024-04-24T22:39:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 719209472 bytes 2024-04-24T22:39:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 759163904 bytes 2024-04-24T22:39:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 802696192 bytes 2024-04-24T22:39:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 827216896 bytes 2024-04-24T22:39:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 872821760 bytes 2024-04-24T22:39:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 900507648 bytes 2024-04-24T22:39:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 961426432 bytes 2024-04-24T22:39:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1001033728 bytes 2024-04-24T22:39:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1033846784 bytes 2024-04-24T22:39:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1054654464 bytes 2024-04-24T22:39:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1108041728 bytes 2024-04-24T22:39:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1149719552 bytes 2024-04-24T22:39:29.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1180894208 bytes 2024-04-24T22:39:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1205020672 bytes 2024-04-24T22:39:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1240783872 bytes 2024-04-24T22:39:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1273765888 bytes 2024-04-24T22:39:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1310115840 bytes 2024-04-24T22:39:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1359780864 bytes 2024-04-24T22:39:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1398621184 bytes 2024-04-24T22:39:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1443496960 bytes 2024-04-24T22:39:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1448643584 bytes 2024-04-24T22:39:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1498242048 bytes 2024-04-24T22:39:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1498846208 bytes 2024-04-24T22:39:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1549647872 bytes 2024-04-24T22:39:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1588166656 bytes 2024-04-24T22:39:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1594433536 bytes 2024-04-24T22:39:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1660611584 bytes 2024-04-24T22:39:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1755694080 bytes 2024-04-24T22:39:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1776035840 bytes 2024-04-24T22:39:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1795565568 bytes 2024-04-24T22:39:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1801374720 bytes 2024-04-24T22:39:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1832895488 bytes 2024-04-24T22:39:32.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1864705024 bytes 2024-04-24T22:39:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 1867537408 bytes 2024-04-24T22:39:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:39:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:39:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:39:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:39:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:39:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_2 2024-04-24T22:39:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2024-04-24T22:39:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:39:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:39:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:39:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_2 128M 2024-04-24T22:39:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:39:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:39:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:39:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:39:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2024-04-24T22:39:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:39:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2024-04-24T22:39:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:39:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:39:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-04-24T22:39:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:39:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-04-24T22:39:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_2': 2024-04-24T22:39:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:39:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:39:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:39:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 127f279dc51e 2024-04-24T22:39:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.127f279dc51e 2024-04-24T22:39:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:39:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:39:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:39:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:39:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:39:32 2024 2024-04-24T22:39:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:39:32 2024 2024-04-24T22:39:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:39:32 2024 2024-04-24T22:39:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:39:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:39:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 9d47d6f5-f040-4915-aac4-a0dd1e04cc47 2024-04-24T22:39:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:39:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:39:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_2 2024-04-24T22:39:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:39:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:39:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=12 2024-04-24T22:39:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:39:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:39:32.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 21493760 bytes 2024-04-24T22:39:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 59559936 bytes 2024-04-24T22:39:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 81269760 bytes 2024-04-24T22:39:33.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 144247808 bytes 2024-04-24T22:39:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 173489152 bytes 2024-04-24T22:39:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 201179136 bytes 2024-04-24T22:39:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 220690432 bytes 2024-04-24T22:39:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 257611776 bytes 2024-04-24T22:39:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 295349248 bytes 2024-04-24T22:39:33.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 305566720 bytes 2024-04-24T22:39:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 367123456 bytes 2024-04-24T22:39:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 408686592 bytes 2024-04-24T22:39:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 427290624 bytes 2024-04-24T22:39:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 486332416 bytes 2024-04-24T22:39:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 547958784 bytes 2024-04-24T22:39:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 559442944 bytes 2024-04-24T22:39:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 588702720 bytes 2024-04-24T22:39:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 591316992 bytes 2024-04-24T22:39:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 671056896 bytes 2024-04-24T22:39:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 682644480 bytes 2024-04-24T22:39:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 689695744 bytes 2024-04-24T22:39:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 720759808 bytes 2024-04-24T22:39:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 825003008 bytes 2024-04-24T22:39:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 851125248 bytes 2024-04-24T22:39:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 880504832 bytes 2024-04-24T22:39:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 936184832 bytes 2024-04-24T22:39:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 957449216 bytes 2024-04-24T22:39:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1013726208 bytes 2024-04-24T22:39:36.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1071302656 bytes 2024-04-24T22:39:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1087258624 bytes 2024-04-24T22:39:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1105086464 bytes 2024-04-24T22:39:36.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1139760128 bytes 2024-04-24T22:39:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1172434944 bytes 2024-04-24T22:39:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1239205888 bytes 2024-04-24T22:39:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1302147072 bytes 2024-04-24T22:39:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1335514112 bytes 2024-04-24T22:39:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1345312768 bytes 2024-04-24T22:39:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1387648000 bytes 2024-04-24T22:39:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1404032000 bytes 2024-04-24T22:39:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1427723264 bytes 2024-04-24T22:39:37.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1471433728 bytes 2024-04-24T22:39:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1483944960 bytes 2024-04-24T22:39:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1498493952 bytes 2024-04-24T22:39:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1537775616 bytes 2024-04-24T22:39:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1607274496 bytes 2024-04-24T22:39:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1607914496 bytes 2024-04-24T22:39:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1629302784 bytes 2024-04-24T22:39:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1663969280 bytes 2024-04-24T22:39:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 1723150336 bytes 2024-04-24T22:39:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 1790360576 bytes 2024-04-24T22:39:38.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 1826339840 bytes 2024-04-24T22:39:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 1877737472 bytes 2024-04-24T22:39:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 1906781184 bytes 2024-04-24T22:39:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 1913022464 bytes 2024-04-24T22:39:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 1961110528 bytes 2024-04-24T22:39:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 1962494976 bytes 2024-04-24T22:39:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 1977409536 bytes 2024-04-24T22:39:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 1978025984 bytes 2024-04-24T22:39:39.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2010906624 bytes 2024-04-24T22:39:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2065504256 bytes 2024-04-24T22:39:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2172108800 bytes 2024-04-24T22:39:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2275225600 bytes 2024-04-24T22:39:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2288276480 bytes 2024-04-24T22:39:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2301598720 bytes 2024-04-24T22:39:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2338172928 bytes 2024-04-24T22:39:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2350954496 bytes 2024-04-24T22:39:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2421324800 bytes 2024-04-24T22:39:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2445929472 bytes 2024-04-24T22:39:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2476965888 bytes 2024-04-24T22:39:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 2560866304 bytes 2024-04-24T22:39:40.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 2586295296 bytes 2024-04-24T22:39:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 2658842624 bytes 2024-04-24T22:39:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 2678296576 bytes 2024-04-24T22:39:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 2738876416 bytes 2024-04-24T22:39:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 2817437696 bytes 2024-04-24T22:39:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 2818065408 bytes 2024-04-24T22:39:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 2868169728 bytes 2024-04-24T22:39:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 2928979968 bytes 2024-04-24T22:39:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7900 IOs, 2971255808 bytes 2024-04-24T22:39:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8000 IOs, 3015901184 bytes 2024-04-24T22:39:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8100 IOs, 3050999808 bytes 2024-04-24T22:39:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8200 IOs, 3076048896 bytes 2024-04-24T22:39:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8300 IOs, 3104171008 bytes 2024-04-24T22:39:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8400 IOs, 3173264384 bytes 2024-04-24T22:39:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8500 IOs, 3247411200 bytes 2024-04-24T22:39:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8600 IOs, 3287780352 bytes 2024-04-24T22:39:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8700 IOs, 3336425472 bytes 2024-04-24T22:39:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8800 IOs, 3403016192 bytes 2024-04-24T22:39:43.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8900 IOs, 3474449408 bytes 2024-04-24T22:39:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9000 IOs, 3530316800 bytes 2024-04-24T22:39:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9100 IOs, 3553087488 bytes 2024-04-24T22:39:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9200 IOs, 3553698816 bytes 2024-04-24T22:39:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9300 IOs, 3584840704 bytes 2024-04-24T22:39:43.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9400 IOs, 3600328704 bytes 2024-04-24T22:39:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9500 IOs, 3672305664 bytes 2024-04-24T22:39:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9600 IOs, 3724795904 bytes 2024-04-24T22:39:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9700 IOs, 3736261632 bytes 2024-04-24T22:39:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9800 IOs, 3763565568 bytes 2024-04-24T22:39:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9900 IOs, 3810664448 bytes 2024-04-24T22:39:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10000 IOs, 3867335680 bytes 2024-04-24T22:39:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10100 IOs, 3877671936 bytes 2024-04-24T22:39:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:10200 IOs, 3917609984 bytes 2024-04-24T22:39:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:39:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:39:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:39:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:39:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:39:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_3 2024-04-24T22:39:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2024-04-24T22:39:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:39:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:39:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:39:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_3 128M 2024-04-24T22:39:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:39:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:39:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:39:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:39:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2024-04-24T22:39:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:39:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2024-04-24T22:39:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:39:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:39:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-04-24T22:39:45.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:39:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-04-24T22:39:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_3': 2024-04-24T22:39:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:39:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:39:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:39:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 128bf01133d4 2024-04-24T22:39:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.128bf01133d4 2024-04-24T22:39:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:39:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:39:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:39:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:39:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:39:44 2024 2024-04-24T22:39:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:39:44 2024 2024-04-24T22:39:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:39:44 2024 2024-04-24T22:39:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:39:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:39:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: e799308a-a04d-4e8c-b6d0-1485ec8486a2 2024-04-24T22:39:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:39:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:39:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_3 2024-04-24T22:39:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:39:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:39:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=6 2024-04-24T22:39:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:39:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:39:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 45697024 bytes 2024-04-24T22:39:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 50442240 bytes 2024-04-24T22:39:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 53665792 bytes 2024-04-24T22:39:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 63186944 bytes 2024-04-24T22:39:45.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 121844736 bytes 2024-04-24T22:39:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 197160960 bytes 2024-04-24T22:39:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 240990208 bytes 2024-04-24T22:39:46.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 268135424 bytes 2024-04-24T22:39:46.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 326513664 bytes 2024-04-24T22:39:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 337741824 bytes 2024-04-24T22:39:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 439231488 bytes 2024-04-24T22:39:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 465179648 bytes 2024-04-24T22:39:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 502724608 bytes 2024-04-24T22:39:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 541750272 bytes 2024-04-24T22:39:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 570822656 bytes 2024-04-24T22:39:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 582164480 bytes 2024-04-24T22:39:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 612633600 bytes 2024-04-24T22:39:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 677010432 bytes 2024-04-24T22:39:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 701122560 bytes 2024-04-24T22:39:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 788409344 bytes 2024-04-24T22:39:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 811127808 bytes 2024-04-24T22:39:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 878678016 bytes 2024-04-24T22:39:48.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 980647936 bytes 2024-04-24T22:39:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1064229888 bytes 2024-04-24T22:39:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1158254592 bytes 2024-04-24T22:39:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1204290560 bytes 2024-04-24T22:39:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1277646848 bytes 2024-04-24T22:39:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1335424000 bytes 2024-04-24T22:39:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1377496064 bytes 2024-04-24T22:39:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1428410368 bytes 2024-04-24T22:39:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1491909632 bytes 2024-04-24T22:39:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1577434112 bytes 2024-04-24T22:39:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1620304896 bytes 2024-04-24T22:39:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1658874880 bytes 2024-04-24T22:39:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1678360576 bytes 2024-04-24T22:39:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1726493696 bytes 2024-04-24T22:39:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1801705472 bytes 2024-04-24T22:39:50.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1890293760 bytes 2024-04-24T22:39:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1903430656 bytes 2024-04-24T22:39:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1929139200 bytes 2024-04-24T22:39:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 2038549504 bytes 2024-04-24T22:39:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 2094305280 bytes 2024-04-24T22:39:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 2126714880 bytes 2024-04-24T22:39:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 2128407552 bytes 2024-04-24T22:39:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 2151305216 bytes 2024-04-24T22:39:51.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 2172157952 bytes 2024-04-24T22:39:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:39:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:39:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:39:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:39:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:39:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_4 2024-04-24T22:39:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2024-04-24T22:39:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:39:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:39:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:39:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_4 128M 2024-04-24T22:39:51.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:51.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:51.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:51.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:51.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:39:51.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:51.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:39:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:39:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:39:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2024-04-24T22:39:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:39:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2024-04-24T22:39:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:39:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:39:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-04-24T22:39:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:39:52.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-04-24T22:39:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_4': 2024-04-24T22:39:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:39:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:39:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:39:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 12975eaac139 2024-04-24T22:39:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.12975eaac139 2024-04-24T22:39:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:39:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:39:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:39:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:39:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:39:51 2024 2024-04-24T22:39:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:39:51 2024 2024-04-24T22:39:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:39:51 2024 2024-04-24T22:39:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:39:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:39:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 455d42c6-ae74-4f65-ae3f-0405438be886 2024-04-24T22:39:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:39:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:39:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_4 2024-04-24T22:39:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:39:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:39:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=5 2024-04-24T22:39:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:39:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:39:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 635904 bytes 2024-04-24T22:39:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 71884800 bytes 2024-04-24T22:39:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 112347136 bytes 2024-04-24T22:39:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 146369536 bytes 2024-04-24T22:39:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 156383232 bytes 2024-04-24T22:39:52.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 198608896 bytes 2024-04-24T22:39:53.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 225035264 bytes 2024-04-24T22:39:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 238242816 bytes 2024-04-24T22:39:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 238888960 bytes 2024-04-24T22:39:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 298313728 bytes 2024-04-24T22:39:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 360004608 bytes 2024-04-24T22:39:53.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 436692992 bytes 2024-04-24T22:39:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 503140352 bytes 2024-04-24T22:39:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 567287808 bytes 2024-04-24T22:39:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 629260288 bytes 2024-04-24T22:39:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 702357504 bytes 2024-04-24T22:39:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 716684288 bytes 2024-04-24T22:39:54.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 757420032 bytes 2024-04-24T22:39:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 845783040 bytes 2024-04-24T22:39:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 874390528 bytes 2024-04-24T22:39:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 933667840 bytes 2024-04-24T22:39:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 1002131456 bytes 2024-04-24T22:39:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1020755968 bytes 2024-04-24T22:39:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1088171008 bytes 2024-04-24T22:39:55.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1122323456 bytes 2024-04-24T22:39:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1161688064 bytes 2024-04-24T22:39:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1167458304 bytes 2024-04-24T22:39:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1173655552 bytes 2024-04-24T22:39:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1174286336 bytes 2024-04-24T22:39:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1195198464 bytes 2024-04-24T22:39:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1219709952 bytes 2024-04-24T22:39:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1228580864 bytes 2024-04-24T22:39:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1305758720 bytes 2024-04-24T22:39:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1337675776 bytes 2024-04-24T22:39:56.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1364221952 bytes 2024-04-24T22:39:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1396177920 bytes 2024-04-24T22:39:57.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1422525440 bytes 2024-04-24T22:39:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1469895680 bytes 2024-04-24T22:39:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:39:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:39:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:39:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:39:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:39:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_5 2024-04-24T22:39:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2024-04-24T22:39:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:39:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:39:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:39:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_5 128M 2024-04-24T22:39:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:39:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:39:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:39:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:39:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:39:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2024-04-24T22:39:57.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:39:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2024-04-24T22:39:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:39:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:39:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-04-24T22:39:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:39:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-04-24T22:39:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_5': 2024-04-24T22:39:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:39:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:39:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:39:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 12a3b9fdc62b 2024-04-24T22:39:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.12a3b9fdc62b 2024-04-24T22:39:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:39:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:39:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:39:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:39:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:39:57 2024 2024-04-24T22:39:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:39:57 2024 2024-04-24T22:39:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:39:57 2024 2024-04-24T22:39:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:39:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:39:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 6fb19b55-9e60-4b51-937c-bfff39ac7c68 2024-04-24T22:39:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:39:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:39:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_5 2024-04-24T22:39:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:39:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:39:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:39:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:39:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=12 2024-04-24T22:39:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:39:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:39:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 37106688 bytes 2024-04-24T22:39:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 87365632 bytes 2024-04-24T22:39:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 152573952 bytes 2024-04-24T22:39:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 201450496 bytes 2024-04-24T22:39:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 218906624 bytes 2024-04-24T22:39:58.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 223672320 bytes 2024-04-24T22:39:59.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 281689088 bytes 2024-04-24T22:39:59.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 332949504 bytes 2024-04-24T22:39:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 409579520 bytes 2024-04-24T22:39:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 448997376 bytes 2024-04-24T22:39:59.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 489590784 bytes 2024-04-24T22:39:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 559729664 bytes 2024-04-24T22:39:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 597452800 bytes 2024-04-24T22:40:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 645850112 bytes 2024-04-24T22:40:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 708665344 bytes 2024-04-24T22:40:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 817283072 bytes 2024-04-24T22:40:00.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 873548800 bytes 2024-04-24T22:40:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 908704768 bytes 2024-04-24T22:40:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 985276416 bytes 2024-04-24T22:40:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 1081582592 bytes 2024-04-24T22:40:01.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 1082192896 bytes 2024-04-24T22:40:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 1120166912 bytes 2024-04-24T22:40:01.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1146137600 bytes 2024-04-24T22:40:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1195329536 bytes 2024-04-24T22:40:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1238545408 bytes 2024-04-24T22:40:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1284044800 bytes 2024-04-24T22:40:01.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1339115520 bytes 2024-04-24T22:40:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1380288512 bytes 2024-04-24T22:40:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1406859264 bytes 2024-04-24T22:40:02.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1430114304 bytes 2024-04-24T22:40:02.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1465785344 bytes 2024-04-24T22:40:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1513643008 bytes 2024-04-24T22:40:02.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1564958720 bytes 2024-04-24T22:40:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1601782784 bytes 2024-04-24T22:40:02.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1610083328 bytes 2024-04-24T22:40:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1660827648 bytes 2024-04-24T22:40:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1708236800 bytes 2024-04-24T22:40:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1751923712 bytes 2024-04-24T22:40:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1776278528 bytes 2024-04-24T22:40:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1796776960 bytes 2024-04-24T22:40:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1819583488 bytes 2024-04-24T22:40:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1888131072 bytes 2024-04-24T22:40:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1904185344 bytes 2024-04-24T22:40:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1914810368 bytes 2024-04-24T22:40:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1927626752 bytes 2024-04-24T22:40:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1957120000 bytes 2024-04-24T22:40:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1968225280 bytes 2024-04-24T22:40:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1998690304 bytes 2024-04-24T22:40:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2035469312 bytes 2024-04-24T22:40:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2036403200 bytes 2024-04-24T22:40:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2077231104 bytes 2024-04-24T22:40:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2131117056 bytes 2024-04-24T22:40:04.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2151141376 bytes 2024-04-24T22:40:04.898 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T22:40:04.896+0000 7fb1e9f85700 -1 reset not still connected to 0x5586bc951ba0 2024-04-24T22:40:04.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2178277376 bytes 2024-04-24T22:40:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2201274368 bytes 2024-04-24T22:40:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2245752832 bytes 2024-04-24T22:40:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2315088896 bytes 2024-04-24T22:40:05.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2366406656 bytes 2024-04-24T22:40:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2405291008 bytes 2024-04-24T22:40:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2415840256 bytes 2024-04-24T22:40:05.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2454272000 bytes 2024-04-24T22:40:05.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2485778432 bytes 2024-04-24T22:40:06.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2533082112 bytes 2024-04-24T22:40:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2547118080 bytes 2024-04-24T22:40:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2595646464 bytes 2024-04-24T22:40:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2711415808 bytes 2024-04-24T22:40:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2719253504 bytes 2024-04-24T22:40:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2755241984 bytes 2024-04-24T22:40:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2780906496 bytes 2024-04-24T22:40:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 2811405312 bytes 2024-04-24T22:40:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 2851947520 bytes 2024-04-24T22:40:07.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 2954500096 bytes 2024-04-24T22:40:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 2989836288 bytes 2024-04-24T22:40:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 3020772352 bytes 2024-04-24T22:40:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 3029022720 bytes 2024-04-24T22:40:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 3078815744 bytes 2024-04-24T22:40:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 3118215168 bytes 2024-04-24T22:40:07.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 3155096576 bytes 2024-04-24T22:40:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7900 IOs, 3182182400 bytes 2024-04-24T22:40:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8000 IOs, 3231682560 bytes 2024-04-24T22:40:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8100 IOs, 3304023040 bytes 2024-04-24T22:40:08.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8200 IOs, 3366987776 bytes 2024-04-24T22:40:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8300 IOs, 3424933888 bytes 2024-04-24T22:40:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8400 IOs, 3455233024 bytes 2024-04-24T22:40:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8500 IOs, 3471755264 bytes 2024-04-24T22:40:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8600 IOs, 3545776128 bytes 2024-04-24T22:40:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8700 IOs, 3560038400 bytes 2024-04-24T22:40:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8800 IOs, 3583735808 bytes 2024-04-24T22:40:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8900 IOs, 3626936320 bytes 2024-04-24T22:40:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9000 IOs, 3718849536 bytes 2024-04-24T22:40:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9100 IOs, 3730512896 bytes 2024-04-24T22:40:09.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9200 IOs, 3806029824 bytes 2024-04-24T22:40:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9300 IOs, 3862646784 bytes 2024-04-24T22:40:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9400 IOs, 3955074048 bytes 2024-04-24T22:40:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:40:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:40:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:40:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:40:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:40:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_6 2024-04-24T22:40:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2024-04-24T22:40:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T22:40:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:40:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:40:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:10.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_6 128M 2024-04-24T22:40:10.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:10.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:10.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:10.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:10.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T22:40:10.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:10.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:40:10.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:40:10.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:40:10.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:10.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2024-04-24T22:40:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:40:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2024-04-24T22:40:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T22:40:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:40:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-04-24T22:40:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:40:11.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-04-24T22:40:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_6': 2024-04-24T22:40:11.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:40:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:40:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:40:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 12af7de4edda 2024-04-24T22:40:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.12af7de4edda 2024-04-24T22:40:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:40:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:40:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:40:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:40:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:40:10 2024 2024-04-24T22:40:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:40:10 2024 2024-04-24T22:40:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:40:10 2024 2024-04-24T22:40:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:40:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:40:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 059ce48e-b70a-434e-9dd3-e955c7a954da 2024-04-24T22:40:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:40:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:40:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_6 2024-04-24T22:40:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T22:40:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:40:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=11 2024-04-24T22:40:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:40:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:40:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 31836160 bytes 2024-04-24T22:40:11.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 90492928 bytes 2024-04-24T22:40:11.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 162535424 bytes 2024-04-24T22:40:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 163172352 bytes 2024-04-24T22:40:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 183816192 bytes 2024-04-24T22:40:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 184424448 bytes 2024-04-24T22:40:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 234283008 bytes 2024-04-24T22:40:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 243018752 bytes 2024-04-24T22:40:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 321421312 bytes 2024-04-24T22:40:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 402721792 bytes 2024-04-24T22:40:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 467337216 bytes 2024-04-24T22:40:12.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 473287680 bytes 2024-04-24T22:40:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 473902080 bytes 2024-04-24T22:40:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 503870464 bytes 2024-04-24T22:40:12.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 536403968 bytes 2024-04-24T22:40:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 570585088 bytes 2024-04-24T22:40:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 621619200 bytes 2024-04-24T22:40:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 622237696 bytes 2024-04-24T22:40:13.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 650470400 bytes 2024-04-24T22:40:13.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 701601792 bytes 2024-04-24T22:40:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 741494784 bytes 2024-04-24T22:40:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 803212288 bytes 2024-04-24T22:40:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 873744384 bytes 2024-04-24T22:40:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 909531136 bytes 2024-04-24T22:40:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 958578688 bytes 2024-04-24T22:40:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 978519040 bytes 2024-04-24T22:40:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1042104320 bytes 2024-04-24T22:40:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1082634240 bytes 2024-04-24T22:40:14.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1101674496 bytes 2024-04-24T22:40:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1130488832 bytes 2024-04-24T22:40:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1209880576 bytes 2024-04-24T22:40:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1217709056 bytes 2024-04-24T22:40:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1239600128 bytes 2024-04-24T22:40:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1267722240 bytes 2024-04-24T22:40:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1285386240 bytes 2024-04-24T22:40:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1361078272 bytes 2024-04-24T22:40:15.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1409316864 bytes 2024-04-24T22:40:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1417010176 bytes 2024-04-24T22:40:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1475397632 bytes 2024-04-24T22:40:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1490819072 bytes 2024-04-24T22:40:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1557764096 bytes 2024-04-24T22:40:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1565181952 bytes 2024-04-24T22:40:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1591764992 bytes 2024-04-24T22:40:16.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1662210048 bytes 2024-04-24T22:40:16.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1691846656 bytes 2024-04-24T22:40:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1718517760 bytes 2024-04-24T22:40:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1757664256 bytes 2024-04-24T22:40:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1830078464 bytes 2024-04-24T22:40:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 1859540992 bytes 2024-04-24T22:40:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 1904771072 bytes 2024-04-24T22:40:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 1936883712 bytes 2024-04-24T22:40:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 1954781184 bytes 2024-04-24T22:40:17.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2009042944 bytes 2024-04-24T22:40:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2037551104 bytes 2024-04-24T22:40:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2052361216 bytes 2024-04-24T22:40:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2076431360 bytes 2024-04-24T22:40:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2121154560 bytes 2024-04-24T22:40:18.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2192577536 bytes 2024-04-24T22:40:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2243531776 bytes 2024-04-24T22:40:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2285708288 bytes 2024-04-24T22:40:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2299252736 bytes 2024-04-24T22:40:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2299867136 bytes 2024-04-24T22:40:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2335035392 bytes 2024-04-24T22:40:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2400307200 bytes 2024-04-24T22:40:19.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2470600704 bytes 2024-04-24T22:40:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2499128320 bytes 2024-04-24T22:40:19.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2531497984 bytes 2024-04-24T22:40:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2561671168 bytes 2024-04-24T22:40:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2644643840 bytes 2024-04-24T22:40:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 2683085824 bytes 2024-04-24T22:40:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 2719905792 bytes 2024-04-24T22:40:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 2784565248 bytes 2024-04-24T22:40:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 2911336448 bytes 2024-04-24T22:40:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 2929401856 bytes 2024-04-24T22:40:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 2930004992 bytes 2024-04-24T22:40:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 3012455424 bytes 2024-04-24T22:40:20.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 3045925888 bytes 2024-04-24T22:40:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 3090795520 bytes 2024-04-24T22:40:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7900 IOs, 3100956672 bytes 2024-04-24T22:40:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8000 IOs, 3101576192 bytes 2024-04-24T22:40:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8100 IOs, 3196120064 bytes 2024-04-24T22:40:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8200 IOs, 3199748096 bytes 2024-04-24T22:40:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8300 IOs, 3200376832 bytes 2024-04-24T22:40:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8400 IOs, 3236719616 bytes 2024-04-24T22:40:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8500 IOs, 3259584512 bytes 2024-04-24T22:40:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:40:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:40:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:40:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:40:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:40:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_7 2024-04-24T22:40:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2024-04-24T22:40:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T22:40:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:40:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:40:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_7 128M 2024-04-24T22:40:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T22:40:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:40:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:40:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:40:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2024-04-24T22:40:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:40:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2024-04-24T22:40:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T22:40:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:40:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-04-24T22:40:22.390 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-24T22:40:22.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-24T22:40:22.391 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-04-24T22:40:22.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:40:22.390+0000 7fa7e233c700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 53474) UID: 1000 2024-04-24T22:40:22.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:40:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:40:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-04-24T22:40:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_7': 2024-04-24T22:40:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:40:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:40:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:40:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 12bb4bb664de 2024-04-24T22:40:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.12bb4bb664de 2024-04-24T22:40:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:40:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:40:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:40:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:40:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:40:22 2024 2024-04-24T22:40:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:40:22 2024 2024-04-24T22:40:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:40:22 2024 2024-04-24T22:40:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:40:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:40:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 3cf64901-3ddb-4c2f-aca5-336fbdf69cfe 2024-04-24T22:40:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:40:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:40:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_7 2024-04-24T22:40:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T22:40:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:40:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=6 2024-04-24T22:40:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:40:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:40:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 14127104 bytes 2024-04-24T22:40:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 74327040 bytes 2024-04-24T22:40:23.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 96813056 bytes 2024-04-24T22:40:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 150209536 bytes 2024-04-24T22:40:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 210745344 bytes 2024-04-24T22:40:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 290395136 bytes 2024-04-24T22:40:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 347301888 bytes 2024-04-24T22:40:24.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 377538560 bytes 2024-04-24T22:40:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 454130688 bytes 2024-04-24T22:40:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 496226304 bytes 2024-04-24T22:40:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 506671104 bytes 2024-04-24T22:40:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 546470912 bytes 2024-04-24T22:40:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 597138432 bytes 2024-04-24T22:40:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 641853440 bytes 2024-04-24T22:40:25.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 705289216 bytes 2024-04-24T22:40:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 713250816 bytes 2024-04-24T22:40:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 726195200 bytes 2024-04-24T22:40:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 791711744 bytes 2024-04-24T22:40:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 831289344 bytes 2024-04-24T22:40:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 881385472 bytes 2024-04-24T22:40:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 957345792 bytes 2024-04-24T22:40:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 962166784 bytes 2024-04-24T22:40:26.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1023135744 bytes 2024-04-24T22:40:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1060864000 bytes 2024-04-24T22:40:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1127635968 bytes 2024-04-24T22:40:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1169378304 bytes 2024-04-24T22:40:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1224795136 bytes 2024-04-24T22:40:26.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1225414656 bytes 2024-04-24T22:40:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1255939072 bytes 2024-04-24T22:40:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1263970304 bytes 2024-04-24T22:40:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1327623168 bytes 2024-04-24T22:40:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1406135296 bytes 2024-04-24T22:40:27.391 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-24T22:40:27.392 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:40:27.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:40:27.392 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-24T22:40:27.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-24T22:40:27.392 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-24T22:40:27.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-24T22:40:27.396 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2024-04-24T22:40:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1436167168 bytes 2024-04-24T22:40:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1492625408 bytes 2024-04-24T22:40:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1551906816 bytes 2024-04-24T22:40:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1631716352 bytes 2024-04-24T22:40:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1677165568 bytes 2024-04-24T22:40:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1766884352 bytes 2024-04-24T22:40:28.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1840461824 bytes 2024-04-24T22:40:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1863798784 bytes 2024-04-24T22:40:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1908801536 bytes 2024-04-24T22:40:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1965861888 bytes 2024-04-24T22:40:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 2027639808 bytes 2024-04-24T22:40:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 2075265024 bytes 2024-04-24T22:40:29.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:40:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:40:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:40:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:40:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:40:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_8 2024-04-24T22:40:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2024-04-24T22:40:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T22:40:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:40:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:40:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_8 128M 2024-04-24T22:40:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T22:40:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:40:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:40:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:40:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2024-04-24T22:40:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:40:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2024-04-24T22:40:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T22:40:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:40:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-04-24T22:40:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:40:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-04-24T22:40:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_8': 2024-04-24T22:40:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:40:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:40:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:40:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 12cf8f434fea 2024-04-24T22:40:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.12cf8f434fea 2024-04-24T22:40:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:40:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:40:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:40:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:40:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:40:29 2024 2024-04-24T22:40:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:40:29 2024 2024-04-24T22:40:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:40:29 2024 2024-04-24T22:40:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:40:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:40:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 8f5f4362-e6f1-477d-bce4-f3ab1a7caee3 2024-04-24T22:40:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:40:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:40:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_8 2024-04-24T22:40:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T22:40:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:40:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=7 2024-04-24T22:40:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:40:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:40:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 29086720 bytes 2024-04-24T22:40:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 65436672 bytes 2024-04-24T22:40:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 105903104 bytes 2024-04-24T22:40:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 185275392 bytes 2024-04-24T22:40:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 185883648 bytes 2024-04-24T22:40:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 221381632 bytes 2024-04-24T22:40:31.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 238317568 bytes 2024-04-24T22:40:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 267931648 bytes 2024-04-24T22:40:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 373872640 bytes 2024-04-24T22:40:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 374498304 bytes 2024-04-24T22:40:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 473091072 bytes 2024-04-24T22:40:31.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 518986752 bytes 2024-04-24T22:40:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 545916928 bytes 2024-04-24T22:40:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 583702528 bytes 2024-04-24T22:40:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 646197248 bytes 2024-04-24T22:40:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 682801152 bytes 2024-04-24T22:40:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 723562496 bytes 2024-04-24T22:40:32.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 765211648 bytes 2024-04-24T22:40:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 772808704 bytes 2024-04-24T22:40:32.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 780281856 bytes 2024-04-24T22:40:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 854397952 bytes 2024-04-24T22:40:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 870516736 bytes 2024-04-24T22:40:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 895293440 bytes 2024-04-24T22:40:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 988155904 bytes 2024-04-24T22:40:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1068105728 bytes 2024-04-24T22:40:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1110687744 bytes 2024-04-24T22:40:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1171573760 bytes 2024-04-24T22:40:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1256354816 bytes 2024-04-24T22:40:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1285687296 bytes 2024-04-24T22:40:34.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1294479360 bytes 2024-04-24T22:40:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1343324160 bytes 2024-04-24T22:40:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1360115712 bytes 2024-04-24T22:40:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1455340544 bytes 2024-04-24T22:40:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1530916864 bytes 2024-04-24T22:40:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1571770368 bytes 2024-04-24T22:40:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1587340288 bytes 2024-04-24T22:40:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1649787904 bytes 2024-04-24T22:40:35.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1759556608 bytes 2024-04-24T22:40:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1821907968 bytes 2024-04-24T22:40:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1851611136 bytes 2024-04-24T22:40:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1887835136 bytes 2024-04-24T22:40:35.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1909761024 bytes 2024-04-24T22:40:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1962602496 bytes 2024-04-24T22:40:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 2057418752 bytes 2024-04-24T22:40:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 2070843392 bytes 2024-04-24T22:40:36.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 2134635520 bytes 2024-04-24T22:40:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 2201709568 bytes 2024-04-24T22:40:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2208754688 bytes 2024-04-24T22:40:36.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2241248256 bytes 2024-04-24T22:40:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2311399424 bytes 2024-04-24T22:40:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2423299072 bytes 2024-04-24T22:40:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:40:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:40:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:40:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:40:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:40:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_9 2024-04-24T22:40:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2024-04-24T22:40:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T22:40:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:40:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:40:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_9 128M 2024-04-24T22:40:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T22:40:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:40:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:40:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:40:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2024-04-24T22:40:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:40:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2024-04-24T22:40:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T22:40:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:40:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-04-24T22:40:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:40:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-04-24T22:40:39.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_9': 2024-04-24T22:40:39.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:40:39.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:40:39.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:40:39.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 12d748220877 2024-04-24T22:40:39.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.12d748220877 2024-04-24T22:40:39.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:40:39.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:40:39.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:40:39.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:40:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:40:37 2024 2024-04-24T22:40:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:40:37 2024 2024-04-24T22:40:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:40:37 2024 2024-04-24T22:40:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:40:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:40:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 59c6ba05-b01f-4e71-ad9c-960605496d0d 2024-04-24T22:40:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:40:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:40:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_9 2024-04-24T22:40:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T22:40:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:40:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=7 2024-04-24T22:40:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:40:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:40:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 21473280 bytes 2024-04-24T22:40:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 82972672 bytes 2024-04-24T22:40:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 84427776 bytes 2024-04-24T22:40:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 124797952 bytes 2024-04-24T22:40:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 128513024 bytes 2024-04-24T22:40:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 232626176 bytes 2024-04-24T22:40:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 274200576 bytes 2024-04-24T22:40:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 346278912 bytes 2024-04-24T22:40:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 360809472 bytes 2024-04-24T22:40:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 430389248 bytes 2024-04-24T22:40:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 438866944 bytes 2024-04-24T22:40:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 493448192 bytes 2024-04-24T22:40:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 544235520 bytes 2024-04-24T22:40:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 642737152 bytes 2024-04-24T22:40:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 705977344 bytes 2024-04-24T22:40:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 773388288 bytes 2024-04-24T22:40:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 813118464 bytes 2024-04-24T22:40:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 883322880 bytes 2024-04-24T22:40:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 902279168 bytes 2024-04-24T22:40:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 935072768 bytes 2024-04-24T22:40:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 1046038528 bytes 2024-04-24T22:40:41.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 1089532928 bytes 2024-04-24T22:40:41.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1117243392 bytes 2024-04-24T22:40:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1197782016 bytes 2024-04-24T22:40:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1250958336 bytes 2024-04-24T22:40:41.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1285307392 bytes 2024-04-24T22:40:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1366843392 bytes 2024-04-24T22:40:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1419785216 bytes 2024-04-24T22:40:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1449921536 bytes 2024-04-24T22:40:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1497123840 bytes 2024-04-24T22:40:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1529776128 bytes 2024-04-24T22:40:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1585627136 bytes 2024-04-24T22:40:42.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1649674240 bytes 2024-04-24T22:40:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1698025472 bytes 2024-04-24T22:40:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1746850816 bytes 2024-04-24T22:40:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1780444160 bytes 2024-04-24T22:40:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1843748864 bytes 2024-04-24T22:40:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1882725376 bytes 2024-04-24T22:40:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1886638080 bytes 2024-04-24T22:40:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1909241856 bytes 2024-04-24T22:40:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1972772864 bytes 2024-04-24T22:40:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 2046452736 bytes 2024-04-24T22:40:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 2061259776 bytes 2024-04-24T22:40:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 2084147200 bytes 2024-04-24T22:40:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 2091754496 bytes 2024-04-24T22:40:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 2192486400 bytes 2024-04-24T22:40:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 2232486912 bytes 2024-04-24T22:40:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2258085888 bytes 2024-04-24T22:40:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2259186688 bytes 2024-04-24T22:40:45.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2307929088 bytes 2024-04-24T22:40:45.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:40:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:40:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:40:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:40:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:40:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_10 2024-04-24T22:40:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2024-04-24T22:40:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T22:40:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:40:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:40:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_10 128M 2024-04-24T22:40:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T22:40:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:40:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:40:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:40:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2024-04-24T22:40:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:40:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2024-04-24T22:40:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T22:40:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:40:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-04-24T22:40:46.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:40:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-04-24T22:40:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_10': 2024-04-24T22:40:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:40:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:40:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:40:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 12df174cba18 2024-04-24T22:40:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.12df174cba18 2024-04-24T22:40:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:40:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:40:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:40:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:40:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:40:45 2024 2024-04-24T22:40:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:40:45 2024 2024-04-24T22:40:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:40:45 2024 2024-04-24T22:40:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:40:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:40:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 020b3f22-a97b-4ec7-aca6-b7942750a5f1 2024-04-24T22:40:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:40:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:40:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_10 2024-04-24T22:40:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T22:40:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:40:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=10 2024-04-24T22:40:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:40:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:40:46.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 13668352 bytes 2024-04-24T22:40:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 31584256 bytes 2024-04-24T22:40:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 149935104 bytes 2024-04-24T22:40:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 172156928 bytes 2024-04-24T22:40:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 234716160 bytes 2024-04-24T22:40:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 288388096 bytes 2024-04-24T22:40:47.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 322610176 bytes 2024-04-24T22:40:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 337053696 bytes 2024-04-24T22:40:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 356859904 bytes 2024-04-24T22:40:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 397906944 bytes 2024-04-24T22:40:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 433140736 bytes 2024-04-24T22:40:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 492418048 bytes 2024-04-24T22:40:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 542098432 bytes 2024-04-24T22:40:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 563392512 bytes 2024-04-24T22:40:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 586549248 bytes 2024-04-24T22:40:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 602463232 bytes 2024-04-24T22:40:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 611599360 bytes 2024-04-24T22:40:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 662994944 bytes 2024-04-24T22:40:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 696835072 bytes 2024-04-24T22:40:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 717407232 bytes 2024-04-24T22:40:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 742402048 bytes 2024-04-24T22:40:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 782324736 bytes 2024-04-24T22:40:49.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 813788160 bytes 2024-04-24T22:40:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 850475008 bytes 2024-04-24T22:40:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 918445056 bytes 2024-04-24T22:40:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 983940096 bytes 2024-04-24T22:40:49.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1049967616 bytes 2024-04-24T22:40:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1109538816 bytes 2024-04-24T22:40:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1110157312 bytes 2024-04-24T22:40:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1150769152 bytes 2024-04-24T22:40:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1163688960 bytes 2024-04-24T22:40:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1221157888 bytes 2024-04-24T22:40:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1253141504 bytes 2024-04-24T22:40:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1278529536 bytes 2024-04-24T22:40:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1328635904 bytes 2024-04-24T22:40:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1363279872 bytes 2024-04-24T22:40:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1388576768 bytes 2024-04-24T22:40:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1449086976 bytes 2024-04-24T22:40:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1470803968 bytes 2024-04-24T22:40:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1526890496 bytes 2024-04-24T22:40:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1536093184 bytes 2024-04-24T22:40:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1536709632 bytes 2024-04-24T22:40:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1551593472 bytes 2024-04-24T22:40:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1570856960 bytes 2024-04-24T22:40:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1623094272 bytes 2024-04-24T22:40:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1649658880 bytes 2024-04-24T22:40:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1667714048 bytes 2024-04-24T22:40:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1689205760 bytes 2024-04-24T22:40:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 1725787136 bytes 2024-04-24T22:40:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 1812835328 bytes 2024-04-24T22:40:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 1835602944 bytes 2024-04-24T22:40:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 1853939712 bytes 2024-04-24T22:40:52.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 1865234432 bytes 2024-04-24T22:40:52.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 1893605376 bytes 2024-04-24T22:40:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 1940286464 bytes 2024-04-24T22:40:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 1979617280 bytes 2024-04-24T22:40:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2029747200 bytes 2024-04-24T22:40:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2032638976 bytes 2024-04-24T22:40:53.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2102830080 bytes 2024-04-24T22:40:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2182396928 bytes 2024-04-24T22:40:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2262342656 bytes 2024-04-24T22:40:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2328988672 bytes 2024-04-24T22:40:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2355917824 bytes 2024-04-24T22:40:54.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2390230016 bytes 2024-04-24T22:40:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2390845440 bytes 2024-04-24T22:40:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2401110016 bytes 2024-04-24T22:40:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2442456064 bytes 2024-04-24T22:40:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2452125696 bytes 2024-04-24T22:40:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2500458496 bytes 2024-04-24T22:40:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 2558577664 bytes 2024-04-24T22:40:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 2606455808 bytes 2024-04-24T22:40:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 2613313536 bytes 2024-04-24T22:40:55.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 2621955072 bytes 2024-04-24T22:40:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 2703471616 bytes 2024-04-24T22:40:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 2716174336 bytes 2024-04-24T22:40:55.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 2730756096 bytes 2024-04-24T22:40:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 2751945728 bytes 2024-04-24T22:40:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 2789657600 bytes 2024-04-24T22:40:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:40:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:40:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:40:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:40:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:40:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_11 2024-04-24T22:40:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2024-04-24T22:40:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T22:40:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:40:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:40:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_11 128M 2024-04-24T22:40:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T22:40:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:40:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:40:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:40:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:40:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_11 2024-04-24T22:40:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:40:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2024-04-24T22:40:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T22:40:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:40:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2024-04-24T22:40:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:40:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2024-04-24T22:40:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_11': 2024-04-24T22:40:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:40:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:40:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:40:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 12e7d305d7c8 2024-04-24T22:40:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.12e7d305d7c8 2024-04-24T22:40:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:40:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:40:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:40:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:40:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:40:56 2024 2024-04-24T22:40:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:40:56 2024 2024-04-24T22:40:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:40:56 2024 2024-04-24T22:40:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:40:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:40:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 8e1cc3c0-2ace-44c5-a115-11e73865c94e 2024-04-24T22:40:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:40:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:40:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_11 2024-04-24T22:40:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:40:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:40:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T22:40:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:40:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=6 2024-04-24T22:40:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:40:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:40:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 47859712 bytes 2024-04-24T22:40:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 117521408 bytes 2024-04-24T22:40:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 157088768 bytes 2024-04-24T22:40:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 221795328 bytes 2024-04-24T22:40:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 258989056 bytes 2024-04-24T22:40:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 325091328 bytes 2024-04-24T22:40:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 359140352 bytes 2024-04-24T22:40:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 386811904 bytes 2024-04-24T22:40:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 443012096 bytes 2024-04-24T22:40:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 493899776 bytes 2024-04-24T22:40:58.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 567168000 bytes 2024-04-24T22:40:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 627686400 bytes 2024-04-24T22:40:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 675528704 bytes 2024-04-24T22:40:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 720978944 bytes 2024-04-24T22:40:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 739417088 bytes 2024-04-24T22:40:59.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 761718784 bytes 2024-04-24T22:40:59.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 802614272 bytes 2024-04-24T22:40:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 846642176 bytes 2024-04-24T22:40:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 912161792 bytes 2024-04-24T22:40:59.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 916664320 bytes 2024-04-24T22:41:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 940506112 bytes 2024-04-24T22:41:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 986144768 bytes 2024-04-24T22:41:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1062606848 bytes 2024-04-24T22:41:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1092548608 bytes 2024-04-24T22:41:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1113811968 bytes 2024-04-24T22:41:00.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1160393728 bytes 2024-04-24T22:41:00.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1166235648 bytes 2024-04-24T22:41:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1193935872 bytes 2024-04-24T22:41:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1269355520 bytes 2024-04-24T22:41:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1326117888 bytes 2024-04-24T22:41:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1356196864 bytes 2024-04-24T22:41:01.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1369469952 bytes 2024-04-24T22:41:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1390974976 bytes 2024-04-24T22:41:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1418533888 bytes 2024-04-24T22:41:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1445831680 bytes 2024-04-24T22:41:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1474501632 bytes 2024-04-24T22:41:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1569344512 bytes 2024-04-24T22:41:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1637081088 bytes 2024-04-24T22:41:02.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1711149056 bytes 2024-04-24T22:41:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1803642880 bytes 2024-04-24T22:41:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1815208960 bytes 2024-04-24T22:41:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1820661760 bytes 2024-04-24T22:41:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1884027904 bytes 2024-04-24T22:41:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1941258240 bytes 2024-04-24T22:41:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:41:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:41:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:41:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:41:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:41:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_12 2024-04-24T22:41:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2024-04-24T22:41:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T22:41:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:41:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:41:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_12 128M 2024-04-24T22:41:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T22:41:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:41:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:41:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:41:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_12 2024-04-24T22:41:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:41:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2024-04-24T22:41:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T22:41:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:41:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2024-04-24T22:41:03.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:41:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2024-04-24T22:41:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_12': 2024-04-24T22:41:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:41:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:41:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:41:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 12f17e6df767 2024-04-24T22:41:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.12f17e6df767 2024-04-24T22:41:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:41:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:41:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:41:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:41:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:41:03 2024 2024-04-24T22:41:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:41:03 2024 2024-04-24T22:41:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:41:03 2024 2024-04-24T22:41:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:41:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:41:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 95e4bc86-7500-4354-96f7-2ccfc5fb3d66 2024-04-24T22:41:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:41:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:41:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_12 2024-04-24T22:41:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T22:41:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:41:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=11 2024-04-24T22:41:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:41:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:41:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 60688384 bytes 2024-04-24T22:41:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 84912128 bytes 2024-04-24T22:41:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 124826624 bytes 2024-04-24T22:41:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 154074112 bytes 2024-04-24T22:41:04.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 215720960 bytes 2024-04-24T22:41:04.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 299265024 bytes 2024-04-24T22:41:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 299887616 bytes 2024-04-24T22:41:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 334455808 bytes 2024-04-24T22:41:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 335059968 bytes 2024-04-24T22:41:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 398510080 bytes 2024-04-24T22:41:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 405213184 bytes 2024-04-24T22:41:05.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 448629760 bytes 2024-04-24T22:41:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 498768896 bytes 2024-04-24T22:41:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 517230592 bytes 2024-04-24T22:41:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 574555136 bytes 2024-04-24T22:41:05.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 640509952 bytes 2024-04-24T22:41:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 647218176 bytes 2024-04-24T22:41:06.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 674104320 bytes 2024-04-24T22:41:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 734624768 bytes 2024-04-24T22:41:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 735250432 bytes 2024-04-24T22:41:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 765631488 bytes 2024-04-24T22:41:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 825573376 bytes 2024-04-24T22:41:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 865585152 bytes 2024-04-24T22:41:06.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 873694208 bytes 2024-04-24T22:41:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 952011776 bytes 2024-04-24T22:41:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 955476992 bytes 2024-04-24T22:41:07.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 969756672 bytes 2024-04-24T22:41:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1017625600 bytes 2024-04-24T22:41:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1042148352 bytes 2024-04-24T22:41:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1074998272 bytes 2024-04-24T22:41:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1118076928 bytes 2024-04-24T22:41:07.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1138300928 bytes 2024-04-24T22:41:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1184684032 bytes 2024-04-24T22:41:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1193590784 bytes 2024-04-24T22:41:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1273183232 bytes 2024-04-24T22:41:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1297844224 bytes 2024-04-24T22:41:08.510 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.2845486144509086) > weight (0.25) 2024-04-24T22:41:08.510 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5720226445447434) > weight (0.25) 2024-04-24T22:41:08.510 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-04-24T22:41:08.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-04-24T22:41:08.510 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-04-24T22:41:08.511 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-04-24T22:41:08.511 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:41:08.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T22:41:08.510+0000 7f1582e1b700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 53264) UID: 1000 2024-04-24T22:41:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1316896768 bytes 2024-04-24T22:41:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1330136064 bytes 2024-04-24T22:41:08.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:41:08.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1378251776 bytes 2024-04-24T22:41:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1399737344 bytes 2024-04-24T22:41:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1535719424 bytes 2024-04-24T22:41:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1542723584 bytes 2024-04-24T22:41:09.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1543346176 bytes 2024-04-24T22:41:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1615346688 bytes 2024-04-24T22:41:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1653491712 bytes 2024-04-24T22:41:09.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1732310016 bytes 2024-04-24T22:41:09.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1749171200 bytes 2024-04-24T22:41:10.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1773668352 bytes 2024-04-24T22:41:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 1806841856 bytes 2024-04-24T22:41:10.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 1857473536 bytes 2024-04-24T22:41:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 1902118912 bytes 2024-04-24T22:41:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 1956336640 bytes 2024-04-24T22:41:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2029822976 bytes 2024-04-24T22:41:10.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2072283136 bytes 2024-04-24T22:41:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2128541696 bytes 2024-04-24T22:41:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2149593088 bytes 2024-04-24T22:41:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2247952384 bytes 2024-04-24T22:41:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2293724160 bytes 2024-04-24T22:41:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2321140736 bytes 2024-04-24T22:41:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2321751040 bytes 2024-04-24T22:41:11.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2345239552 bytes 2024-04-24T22:41:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2374139904 bytes 2024-04-24T22:41:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2412582912 bytes 2024-04-24T22:41:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2455596032 bytes 2024-04-24T22:41:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2497603584 bytes 2024-04-24T22:41:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2518512640 bytes 2024-04-24T22:41:12.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2567178240 bytes 2024-04-24T22:41:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2601894912 bytes 2024-04-24T22:41:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2620395520 bytes 2024-04-24T22:41:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 2669419520 bytes 2024-04-24T22:41:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 2695493632 bytes 2024-04-24T22:41:13.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 2737989632 bytes 2024-04-24T22:41:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 2791365632 bytes 2024-04-24T22:41:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 2791966720 bytes 2024-04-24T22:41:13.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 2850971648 bytes 2024-04-24T22:41:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 2859213824 bytes 2024-04-24T22:41:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 2924922880 bytes 2024-04-24T22:41:13.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 2979200000 bytes 2024-04-24T22:41:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7900 IOs, 3013694464 bytes 2024-04-24T22:41:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8000 IOs, 3061189632 bytes 2024-04-24T22:41:14.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8100 IOs, 3097022464 bytes 2024-04-24T22:41:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8200 IOs, 3173424128 bytes 2024-04-24T22:41:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8300 IOs, 3189345280 bytes 2024-04-24T22:41:14.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:41:14.612 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-04-24T22:41:14.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-24T22:41:14.612 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-24T22:41:14.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-24T22:41:14.629 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2024-04-24T22:41:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8400 IOs, 3223605248 bytes 2024-04-24T22:41:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:41:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:41:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:41:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:41:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:41:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_13 2024-04-24T22:41:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2024-04-24T22:41:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T22:41:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:41:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:41:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_13 128M 2024-04-24T22:41:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T22:41:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:41:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:41:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:41:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_13 2024-04-24T22:41:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:41:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2024-04-24T22:41:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T22:41:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:41:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2024-04-24T22:41:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:41:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2024-04-24T22:41:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_13': 2024-04-24T22:41:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:41:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:41:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:41:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 12fd51be2a08 2024-04-24T22:41:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.12fd51be2a08 2024-04-24T22:41:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:41:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:41:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:41:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:41:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:41:14 2024 2024-04-24T22:41:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:41:14 2024 2024-04-24T22:41:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:41:14 2024 2024-04-24T22:41:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:41:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:41:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 4bd2c65f-f68d-4a16-b6d4-a7ab70837e25 2024-04-24T22:41:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:41:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:41:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_13 2024-04-24T22:41:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T22:41:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:41:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=12 2024-04-24T22:41:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:41:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:41:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 39329792 bytes 2024-04-24T22:41:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 71151616 bytes 2024-04-24T22:41:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 102363136 bytes 2024-04-24T22:41:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 200218624 bytes 2024-04-24T22:41:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 269016064 bytes 2024-04-24T22:41:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 331708416 bytes 2024-04-24T22:41:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 343842816 bytes 2024-04-24T22:41:16.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 366112768 bytes 2024-04-24T22:41:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 402608128 bytes 2024-04-24T22:41:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 442961920 bytes 2024-04-24T22:41:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 486334464 bytes 2024-04-24T22:41:16.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 497274880 bytes 2024-04-24T22:41:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 540554240 bytes 2024-04-24T22:41:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 577468416 bytes 2024-04-24T22:41:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 658324480 bytes 2024-04-24T22:41:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 660469760 bytes 2024-04-24T22:41:17.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 664971264 bytes 2024-04-24T22:41:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 676494336 bytes 2024-04-24T22:41:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 697954304 bytes 2024-04-24T22:41:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 784492544 bytes 2024-04-24T22:41:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 810128384 bytes 2024-04-24T22:41:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 848169984 bytes 2024-04-24T22:41:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 917128192 bytes 2024-04-24T22:41:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 966013952 bytes 2024-04-24T22:41:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1025150976 bytes 2024-04-24T22:41:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1043804160 bytes 2024-04-24T22:41:18.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1080989696 bytes 2024-04-24T22:41:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1134148608 bytes 2024-04-24T22:41:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1194434560 bytes 2024-04-24T22:41:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1236514816 bytes 2024-04-24T22:41:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1278408704 bytes 2024-04-24T22:41:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1336044544 bytes 2024-04-24T22:41:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1384180736 bytes 2024-04-24T22:41:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1426158592 bytes 2024-04-24T22:41:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1434664960 bytes 2024-04-24T22:41:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1493213184 bytes 2024-04-24T22:41:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1506838528 bytes 2024-04-24T22:41:20.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1561074688 bytes 2024-04-24T22:41:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1594473472 bytes 2024-04-24T22:41:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1618076672 bytes 2024-04-24T22:41:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1637331968 bytes 2024-04-24T22:41:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1654936576 bytes 2024-04-24T22:41:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1695579136 bytes 2024-04-24T22:41:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1755398144 bytes 2024-04-24T22:41:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1776870400 bytes 2024-04-24T22:41:21.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1815536640 bytes 2024-04-24T22:41:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1834262528 bytes 2024-04-24T22:41:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1916335104 bytes 2024-04-24T22:41:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 1960972288 bytes 2024-04-24T22:41:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 1980798976 bytes 2024-04-24T22:41:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2034306048 bytes 2024-04-24T22:41:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2099870720 bytes 2024-04-24T22:41:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2128067584 bytes 2024-04-24T22:41:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2142736384 bytes 2024-04-24T22:41:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2168861696 bytes 2024-04-24T22:41:22.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2239326208 bytes 2024-04-24T22:41:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2275317760 bytes 2024-04-24T22:41:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2322758656 bytes 2024-04-24T22:41:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2374420480 bytes 2024-04-24T22:41:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2401725440 bytes 2024-04-24T22:41:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2464880640 bytes 2024-04-24T22:41:23.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2488072192 bytes 2024-04-24T22:41:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2497747968 bytes 2024-04-24T22:41:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2522536960 bytes 2024-04-24T22:41:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2576017408 bytes 2024-04-24T22:41:24.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2623894528 bytes 2024-04-24T22:41:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2702973952 bytes 2024-04-24T22:41:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2779714560 bytes 2024-04-24T22:41:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2794064896 bytes 2024-04-24T22:41:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 2829987840 bytes 2024-04-24T22:41:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 2950307840 bytes 2024-04-24T22:41:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 2968580096 bytes 2024-04-24T22:41:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 3036619776 bytes 2024-04-24T22:41:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 3082227712 bytes 2024-04-24T22:41:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 3149679616 bytes 2024-04-24T22:41:25.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 3162928128 bytes 2024-04-24T22:41:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 3163573248 bytes 2024-04-24T22:41:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 3188256768 bytes 2024-04-24T22:41:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7900 IOs, 3240961024 bytes 2024-04-24T22:41:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8000 IOs, 3292900352 bytes 2024-04-24T22:41:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8100 IOs, 3314530304 bytes 2024-04-24T22:41:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8200 IOs, 3363726336 bytes 2024-04-24T22:41:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8300 IOs, 3410434048 bytes 2024-04-24T22:41:26.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8400 IOs, 3417809920 bytes 2024-04-24T22:41:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8500 IOs, 3467503616 bytes 2024-04-24T22:41:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8600 IOs, 3491330048 bytes 2024-04-24T22:41:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8700 IOs, 3550012416 bytes 2024-04-24T22:41:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:41:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:41:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:41:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:41:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:41:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_14 2024-04-24T22:41:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2024-04-24T22:41:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T22:41:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:41:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:41:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_14 128M 2024-04-24T22:41:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T22:41:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:41:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:41:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:41:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_14 2024-04-24T22:41:27.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:41:27.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2024-04-24T22:41:27.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:27.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:27.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T22:41:27.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:41:27.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2024-04-24T22:41:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:41:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2024-04-24T22:41:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_14': 2024-04-24T22:41:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:41:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:41:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:41:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 130cca3e6e0f 2024-04-24T22:41:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.130cca3e6e0f 2024-04-24T22:41:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:41:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:41:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:41:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:41:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:41:27 2024 2024-04-24T22:41:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:41:27 2024 2024-04-24T22:41:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:41:27 2024 2024-04-24T22:41:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:41:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:41:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: e99fde4e-a334-4491-9ca4-d28b703e6700 2024-04-24T22:41:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:41:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:41:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_14 2024-04-24T22:41:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T22:41:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:41:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=6 2024-04-24T22:41:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:41:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:41:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 26369024 bytes 2024-04-24T22:41:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 94454784 bytes 2024-04-24T22:41:28.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 111647744 bytes 2024-04-24T22:41:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 132988928 bytes 2024-04-24T22:41:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 195620864 bytes 2024-04-24T22:41:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 246292480 bytes 2024-04-24T22:41:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 269140992 bytes 2024-04-24T22:41:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 287925248 bytes 2024-04-24T22:41:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 302721024 bytes 2024-04-24T22:41:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 362402816 bytes 2024-04-24T22:41:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 365104128 bytes 2024-04-24T22:41:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 367850496 bytes 2024-04-24T22:41:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 427081728 bytes 2024-04-24T22:41:29.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 481429504 bytes 2024-04-24T22:41:30.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 503956480 bytes 2024-04-24T22:41:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 554058752 bytes 2024-04-24T22:41:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 577929216 bytes 2024-04-24T22:41:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 620164096 bytes 2024-04-24T22:41:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 666917888 bytes 2024-04-24T22:41:30.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 751293440 bytes 2024-04-24T22:41:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 813409280 bytes 2024-04-24T22:41:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 836239360 bytes 2024-04-24T22:41:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 896302080 bytes 2024-04-24T22:41:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 977265664 bytes 2024-04-24T22:41:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 991336448 bytes 2024-04-24T22:41:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1042117632 bytes 2024-04-24T22:41:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1096653824 bytes 2024-04-24T22:41:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1129528320 bytes 2024-04-24T22:41:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1153254400 bytes 2024-04-24T22:41:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1214624768 bytes 2024-04-24T22:41:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1240008704 bytes 2024-04-24T22:41:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1294800896 bytes 2024-04-24T22:41:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1331741696 bytes 2024-04-24T22:41:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1332358144 bytes 2024-04-24T22:41:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1450102784 bytes 2024-04-24T22:41:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1512578048 bytes 2024-04-24T22:41:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1549646848 bytes 2024-04-24T22:41:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1579359232 bytes 2024-04-24T22:41:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1611021312 bytes 2024-04-24T22:41:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1632506880 bytes 2024-04-24T22:41:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1682847744 bytes 2024-04-24T22:41:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1720352768 bytes 2024-04-24T22:41:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1747153920 bytes 2024-04-24T22:41:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1753541632 bytes 2024-04-24T22:41:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1778269184 bytes 2024-04-24T22:41:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1814592512 bytes 2024-04-24T22:41:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:41:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:41:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:41:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:41:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:41:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_15 2024-04-24T22:41:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2024-04-24T22:41:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T22:41:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:41:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:41:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_15 128M 2024-04-24T22:41:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T22:41:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:41:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:41:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:41:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_15 2024-04-24T22:41:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:41:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2024-04-24T22:41:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T22:41:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:41:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2024-04-24T22:41:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:41:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2024-04-24T22:41:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_15': 2024-04-24T22:41:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:41:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:41:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:41:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 1318103ded53 2024-04-24T22:41:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.1318103ded53 2024-04-24T22:41:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:41:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:41:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:41:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:41:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:41:34 2024 2024-04-24T22:41:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:41:34 2024 2024-04-24T22:41:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:41:34 2024 2024-04-24T22:41:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:41:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:41:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 61a0be27-386b-497f-bf17-b0030cfd9593 2024-04-24T22:41:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:41:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:41:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_15 2024-04-24T22:41:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T22:41:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:41:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=10 2024-04-24T22:41:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:41:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:41:35.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 51663872 bytes 2024-04-24T22:41:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 79129600 bytes 2024-04-24T22:41:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 124513280 bytes 2024-04-24T22:41:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 141372416 bytes 2024-04-24T22:41:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 223960064 bytes 2024-04-24T22:41:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 259337216 bytes 2024-04-24T22:41:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 321391616 bytes 2024-04-24T22:41:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 350116864 bytes 2024-04-24T22:41:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 374860800 bytes 2024-04-24T22:41:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 417560576 bytes 2024-04-24T22:41:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 511954944 bytes 2024-04-24T22:41:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 525093888 bytes 2024-04-24T22:41:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 625520640 bytes 2024-04-24T22:41:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 705024000 bytes 2024-04-24T22:41:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 733536256 bytes 2024-04-24T22:41:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 764790784 bytes 2024-04-24T22:41:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 796051456 bytes 2024-04-24T22:41:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 836376576 bytes 2024-04-24T22:41:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 863047680 bytes 2024-04-24T22:41:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 864709632 bytes 2024-04-24T22:41:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 922778624 bytes 2024-04-24T22:41:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 955309056 bytes 2024-04-24T22:41:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 972819456 bytes 2024-04-24T22:41:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1016093696 bytes 2024-04-24T22:41:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1040310272 bytes 2024-04-24T22:41:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1081049088 bytes 2024-04-24T22:41:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1138052096 bytes 2024-04-24T22:41:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1217446912 bytes 2024-04-24T22:41:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1283015680 bytes 2024-04-24T22:41:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1341852672 bytes 2024-04-24T22:41:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1361950720 bytes 2024-04-24T22:41:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1467108352 bytes 2024-04-24T22:41:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1495534592 bytes 2024-04-24T22:41:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1546219520 bytes 2024-04-24T22:41:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1591118848 bytes 2024-04-24T22:41:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1653702656 bytes 2024-04-24T22:41:40.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1749977088 bytes 2024-04-24T22:41:40.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1784550400 bytes 2024-04-24T22:41:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1860276224 bytes 2024-04-24T22:41:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1925137408 bytes 2024-04-24T22:41:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1961144320 bytes 2024-04-24T22:41:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1984830464 bytes 2024-04-24T22:41:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 2045527040 bytes 2024-04-24T22:41:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 2094087168 bytes 2024-04-24T22:41:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 2162355200 bytes 2024-04-24T22:41:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 2201108480 bytes 2024-04-24T22:41:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 2218944512 bytes 2024-04-24T22:41:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2247713792 bytes 2024-04-24T22:41:42.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2296082432 bytes 2024-04-24T22:41:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2352910336 bytes 2024-04-24T22:41:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2386287616 bytes 2024-04-24T22:41:42.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2438849536 bytes 2024-04-24T22:41:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2475300864 bytes 2024-04-24T22:41:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2518589440 bytes 2024-04-24T22:41:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2539630592 bytes 2024-04-24T22:41:43.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2595943424 bytes 2024-04-24T22:41:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2629605376 bytes 2024-04-24T22:41:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2673970176 bytes 2024-04-24T22:41:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2674590720 bytes 2024-04-24T22:41:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2728135680 bytes 2024-04-24T22:41:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2733981696 bytes 2024-04-24T22:41:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2809646080 bytes 2024-04-24T22:41:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2858164224 bytes 2024-04-24T22:41:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2875030528 bytes 2024-04-24T22:41:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2897244160 bytes 2024-04-24T22:41:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2911826944 bytes 2024-04-24T22:41:44.397 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-24T22:41:44.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-24T22:41:44.397 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-04-24T22:41:44.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:41:44.396+0000 7f3c0b2c1700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 57497) UID: 1000 2024-04-24T22:41:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2946838528 bytes 2024-04-24T22:41:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2993658880 bytes 2024-04-24T22:41:44.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:41:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 3013101568 bytes 2024-04-24T22:41:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 3113924608 bytes 2024-04-24T22:41:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 3167616000 bytes 2024-04-24T22:41:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:41:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:41:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:41:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:41:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:41:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_16 2024-04-24T22:41:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2024-04-24T22:41:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T22:41:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:41:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:41:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_16 128M 2024-04-24T22:41:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T22:41:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:41:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:41:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:41:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_16 2024-04-24T22:41:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:41:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2024-04-24T22:41:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T22:41:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:41:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2024-04-24T22:41:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:41:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2024-04-24T22:41:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_16': 2024-04-24T22:41:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:41:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:41:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:41:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 1324a0681439 2024-04-24T22:41:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.1324a0681439 2024-04-24T22:41:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:41:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:41:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:41:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:41:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:41:45 2024 2024-04-24T22:41:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:41:45 2024 2024-04-24T22:41:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:41:45 2024 2024-04-24T22:41:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:41:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:41:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 73832a9a-0532-4696-9e85-d0aaea380e6c 2024-04-24T22:41:46.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:41:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:41:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_16 2024-04-24T22:41:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T22:41:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:41:46.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=5 2024-04-24T22:41:46.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:41:46.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:41:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 28643328 bytes 2024-04-24T22:41:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 92254208 bytes 2024-04-24T22:41:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 179891200 bytes 2024-04-24T22:41:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 244848640 bytes 2024-04-24T22:41:47.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 252676096 bytes 2024-04-24T22:41:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 324295680 bytes 2024-04-24T22:41:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 329330688 bytes 2024-04-24T22:41:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 378201088 bytes 2024-04-24T22:41:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 494283776 bytes 2024-04-24T22:41:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 513809408 bytes 2024-04-24T22:41:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 558758912 bytes 2024-04-24T22:41:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 599756800 bytes 2024-04-24T22:41:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 644561920 bytes 2024-04-24T22:41:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 692273152 bytes 2024-04-24T22:41:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 798968832 bytes 2024-04-24T22:41:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 893204480 bytes 2024-04-24T22:41:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 947178496 bytes 2024-04-24T22:41:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 983246848 bytes 2024-04-24T22:41:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 1079929856 bytes 2024-04-24T22:41:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 1118964736 bytes 2024-04-24T22:41:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 1134758912 bytes 2024-04-24T22:41:49.398 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-24T22:41:49.398 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:41:49.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:41:49.399 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-24T22:41:49.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-24T22:41:49.400 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-24T22:41:49.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-24T22:41:49.405 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2024-04-24T22:41:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 1135352832 bytes 2024-04-24T22:41:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1204353024 bytes 2024-04-24T22:41:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1207265280 bytes 2024-04-24T22:41:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1231643648 bytes 2024-04-24T22:41:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1232258048 bytes 2024-04-24T22:41:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1282707456 bytes 2024-04-24T22:41:50.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1293185024 bytes 2024-04-24T22:41:50.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1345844224 bytes 2024-04-24T22:41:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1389676544 bytes 2024-04-24T22:41:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1493428224 bytes 2024-04-24T22:41:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1517896704 bytes 2024-04-24T22:41:51.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1561122816 bytes 2024-04-24T22:41:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1606277120 bytes 2024-04-24T22:41:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:41:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:41:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:41:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:41:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:41:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_17 2024-04-24T22:41:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2024-04-24T22:41:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T22:41:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:41:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:41:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_17 128M 2024-04-24T22:41:51.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:51.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:51.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:51.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:51.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T22:41:51.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:51.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:41:51.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:41:51.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:41:51.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:41:51.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_17 2024-04-24T22:41:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:41:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2024-04-24T22:41:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T22:41:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:41:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2024-04-24T22:41:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:41:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2024-04-24T22:41:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_17': 2024-04-24T22:41:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:41:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:41:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:41:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 1339eeb2d329 2024-04-24T22:41:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.1339eeb2d329 2024-04-24T22:41:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:41:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:41:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:41:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:41:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:41:51 2024 2024-04-24T22:41:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:41:51 2024 2024-04-24T22:41:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:41:51 2024 2024-04-24T22:41:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:41:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:41:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: ea68e336-4843-42cb-80ec-9f3fa4c17a25 2024-04-24T22:41:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:41:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:41:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_17 2024-04-24T22:41:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:41:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:41:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T22:41:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:41:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=13 2024-04-24T22:41:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:41:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:41:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 48570368 bytes 2024-04-24T22:41:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 76345344 bytes 2024-04-24T22:41:52.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 83629056 bytes 2024-04-24T22:41:52.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 133394432 bytes 2024-04-24T22:41:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 147001344 bytes 2024-04-24T22:41:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 248228864 bytes 2024-04-24T22:41:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 284953600 bytes 2024-04-24T22:41:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 327485440 bytes 2024-04-24T22:41:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 363040768 bytes 2024-04-24T22:41:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 383267840 bytes 2024-04-24T22:41:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 438222848 bytes 2024-04-24T22:41:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 464416768 bytes 2024-04-24T22:41:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 479181824 bytes 2024-04-24T22:41:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 568719360 bytes 2024-04-24T22:41:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 641285120 bytes 2024-04-24T22:41:54.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 685937664 bytes 2024-04-24T22:41:54.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 706043904 bytes 2024-04-24T22:41:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 811824128 bytes 2024-04-24T22:41:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 856323072 bytes 2024-04-24T22:41:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 912243712 bytes 2024-04-24T22:41:55.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 923197440 bytes 2024-04-24T22:41:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 933943296 bytes 2024-04-24T22:41:55.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1003710464 bytes 2024-04-24T22:41:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1134118912 bytes 2024-04-24T22:41:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1163765760 bytes 2024-04-24T22:41:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1238345728 bytes 2024-04-24T22:41:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1320624128 bytes 2024-04-24T22:41:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1349292032 bytes 2024-04-24T22:41:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1401366528 bytes 2024-04-24T22:41:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1418351616 bytes 2024-04-24T22:41:56.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1515295744 bytes 2024-04-24T22:41:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1525990400 bytes 2024-04-24T22:41:57.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1534338048 bytes 2024-04-24T22:41:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1574806528 bytes 2024-04-24T22:41:57.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1578436608 bytes 2024-04-24T22:41:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1623051264 bytes 2024-04-24T22:41:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1704926208 bytes 2024-04-24T22:41:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1713245184 bytes 2024-04-24T22:41:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1784121344 bytes 2024-04-24T22:41:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1851091968 bytes 2024-04-24T22:41:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1862227968 bytes 2024-04-24T22:41:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1890451456 bytes 2024-04-24T22:41:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1912025088 bytes 2024-04-24T22:41:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1960312832 bytes 2024-04-24T22:41:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1990131712 bytes 2024-04-24T22:41:58.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 2001003520 bytes 2024-04-24T22:41:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 2006992896 bytes 2024-04-24T22:41:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2028344320 bytes 2024-04-24T22:41:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2029878272 bytes 2024-04-24T22:41:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2083680256 bytes 2024-04-24T22:41:59.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2100616192 bytes 2024-04-24T22:41:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2147977216 bytes 2024-04-24T22:41:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2196748288 bytes 2024-04-24T22:41:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2235651072 bytes 2024-04-24T22:41:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2297281536 bytes 2024-04-24T22:41:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2316121088 bytes 2024-04-24T22:42:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2355584000 bytes 2024-04-24T22:42:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2399565824 bytes 2024-04-24T22:42:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2484642816 bytes 2024-04-24T22:42:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2506116096 bytes 2024-04-24T22:42:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2506727424 bytes 2024-04-24T22:42:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2542635008 bytes 2024-04-24T22:42:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2567743488 bytes 2024-04-24T22:42:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2591115264 bytes 2024-04-24T22:42:01.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2607045632 bytes 2024-04-24T22:42:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2637712384 bytes 2024-04-24T22:42:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2673235968 bytes 2024-04-24T22:42:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2756441088 bytes 2024-04-24T22:42:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2798959616 bytes 2024-04-24T22:42:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 2855196672 bytes 2024-04-24T22:42:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 2912846848 bytes 2024-04-24T22:42:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 2928701440 bytes 2024-04-24T22:42:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 2946039808 bytes 2024-04-24T22:42:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 2958991360 bytes 2024-04-24T22:42:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 3006459904 bytes 2024-04-24T22:42:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 3032628224 bytes 2024-04-24T22:42:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 3071304704 bytes 2024-04-24T22:42:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 3105567744 bytes 2024-04-24T22:42:02.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7900 IOs, 3160943616 bytes 2024-04-24T22:42:03.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8000 IOs, 3235591168 bytes 2024-04-24T22:42:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8100 IOs, 3257136128 bytes 2024-04-24T22:42:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8200 IOs, 3285234688 bytes 2024-04-24T22:42:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8300 IOs, 3296758784 bytes 2024-04-24T22:42:03.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8400 IOs, 3306635264 bytes 2024-04-24T22:42:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8500 IOs, 3370071040 bytes 2024-04-24T22:42:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8600 IOs, 3396996096 bytes 2024-04-24T22:42:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8700 IOs, 3428722688 bytes 2024-04-24T22:42:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8800 IOs, 3450560512 bytes 2024-04-24T22:42:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8900 IOs, 3478337536 bytes 2024-04-24T22:42:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9000 IOs, 3478932480 bytes 2024-04-24T22:42:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9100 IOs, 3526987776 bytes 2024-04-24T22:42:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9200 IOs, 3585601536 bytes 2024-04-24T22:42:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9300 IOs, 3661113344 bytes 2024-04-24T22:42:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9400 IOs, 3730267136 bytes 2024-04-24T22:42:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9500 IOs, 3814192128 bytes 2024-04-24T22:42:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:42:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:42:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:42:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:42:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:42:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_18 2024-04-24T22:42:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2024-04-24T22:42:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T22:42:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:42:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:42:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_18 128M 2024-04-24T22:42:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T22:42:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:42:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:42:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:42:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_18 2024-04-24T22:42:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:42:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2024-04-24T22:42:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T22:42:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:42:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2024-04-24T22:42:06.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:42:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2024-04-24T22:42:06.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_18': 2024-04-24T22:42:06.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:42:06.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:42:06.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:42:06.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 1345ad3aa464 2024-04-24T22:42:06.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.1345ad3aa464 2024-04-24T22:42:06.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:42:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:42:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:42:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:42:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:42:05 2024 2024-04-24T22:42:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:42:05 2024 2024-04-24T22:42:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:42:05 2024 2024-04-24T22:42:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:42:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:42:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: e52f7b2d-1d65-436d-bbae-a4af8a5d1482 2024-04-24T22:42:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:42:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:42:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_18 2024-04-24T22:42:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T22:42:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:42:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=6 2024-04-24T22:42:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:42:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:42:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 45767680 bytes 2024-04-24T22:42:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 100812800 bytes 2024-04-24T22:42:06.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 133807104 bytes 2024-04-24T22:42:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 152437760 bytes 2024-04-24T22:42:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 170989568 bytes 2024-04-24T22:42:07.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 255195136 bytes 2024-04-24T22:42:07.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 286638080 bytes 2024-04-24T22:42:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 340208640 bytes 2024-04-24T22:42:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 379121664 bytes 2024-04-24T22:42:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 426397696 bytes 2024-04-24T22:42:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 427008000 bytes 2024-04-24T22:42:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 442263552 bytes 2024-04-24T22:42:08.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 468336640 bytes 2024-04-24T22:42:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 491698176 bytes 2024-04-24T22:42:08.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 525859840 bytes 2024-04-24T22:42:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 528161792 bytes 2024-04-24T22:42:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 578022400 bytes 2024-04-24T22:42:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 634121216 bytes 2024-04-24T22:42:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 664099840 bytes 2024-04-24T22:42:09.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 717400064 bytes 2024-04-24T22:42:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 718015488 bytes 2024-04-24T22:42:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 749974528 bytes 2024-04-24T22:42:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 799280128 bytes 2024-04-24T22:42:09.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 852295680 bytes 2024-04-24T22:42:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 906500096 bytes 2024-04-24T22:42:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 919121920 bytes 2024-04-24T22:42:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1017453568 bytes 2024-04-24T22:42:10.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1031246848 bytes 2024-04-24T22:42:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1075150848 bytes 2024-04-24T22:42:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1141399552 bytes 2024-04-24T22:42:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1159469056 bytes 2024-04-24T22:42:10.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1160046592 bytes 2024-04-24T22:42:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1238018048 bytes 2024-04-24T22:42:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1296717824 bytes 2024-04-24T22:42:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1417440256 bytes 2024-04-24T22:42:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1446599680 bytes 2024-04-24T22:42:11.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1468758016 bytes 2024-04-24T22:42:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1513872384 bytes 2024-04-24T22:42:11.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1569173504 bytes 2024-04-24T22:42:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1592436736 bytes 2024-04-24T22:42:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1597189120 bytes 2024-04-24T22:42:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1673641984 bytes 2024-04-24T22:42:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1751743488 bytes 2024-04-24T22:42:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:42:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:42:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:42:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:42:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:42:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_19 2024-04-24T22:42:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2024-04-24T22:42:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:12.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T22:42:12.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:12.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:42:12.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:42:12.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_19 128M 2024-04-24T22:42:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T22:42:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:42:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:42:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:42:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_19 2024-04-24T22:42:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:42:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2024-04-24T22:42:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T22:42:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:42:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2024-04-24T22:42:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:42:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2024-04-24T22:42:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_19': 2024-04-24T22:42:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:42:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:42:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:42:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 1351c86d44f6 2024-04-24T22:42:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.1351c86d44f6 2024-04-24T22:42:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:42:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:42:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:42:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:42:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:42:12 2024 2024-04-24T22:42:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:42:12 2024 2024-04-24T22:42:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:42:12 2024 2024-04-24T22:42:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:42:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:42:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 38210526-ffd7-4449-be03-5e75b6c23176 2024-04-24T22:42:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:42:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:42:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_19 2024-04-24T22:42:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T22:42:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:42:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=7 2024-04-24T22:42:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:42:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:42:13.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 44494848 bytes 2024-04-24T22:42:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 92673024 bytes 2024-04-24T22:42:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 112462848 bytes 2024-04-24T22:42:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 151108608 bytes 2024-04-24T22:42:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 151698432 bytes 2024-04-24T22:42:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 202134528 bytes 2024-04-24T22:42:14.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 280997888 bytes 2024-04-24T22:42:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 306012160 bytes 2024-04-24T22:42:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 360953856 bytes 2024-04-24T22:42:14.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 381723648 bytes 2024-04-24T22:42:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 437207040 bytes 2024-04-24T22:42:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 475504640 bytes 2024-04-24T22:42:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 526384128 bytes 2024-04-24T22:42:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 545340416 bytes 2024-04-24T22:42:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 565080064 bytes 2024-04-24T22:42:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 608799744 bytes 2024-04-24T22:42:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 691903488 bytes 2024-04-24T22:42:15.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 712342528 bytes 2024-04-24T22:42:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 714112000 bytes 2024-04-24T22:42:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 743889920 bytes 2024-04-24T22:42:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 781731840 bytes 2024-04-24T22:42:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 872175616 bytes 2024-04-24T22:42:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 901514240 bytes 2024-04-24T22:42:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 944039936 bytes 2024-04-24T22:42:16.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1014251520 bytes 2024-04-24T22:42:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1029762048 bytes 2024-04-24T22:42:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1094106112 bytes 2024-04-24T22:42:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1109960704 bytes 2024-04-24T22:42:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1129893888 bytes 2024-04-24T22:42:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1172103168 bytes 2024-04-24T22:42:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1234702336 bytes 2024-04-24T22:42:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1271950336 bytes 2024-04-24T22:42:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1304272896 bytes 2024-04-24T22:42:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1341309952 bytes 2024-04-24T22:42:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1385524224 bytes 2024-04-24T22:42:18.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1456158720 bytes 2024-04-24T22:42:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1498088448 bytes 2024-04-24T22:42:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1542603776 bytes 2024-04-24T22:42:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1543355392 bytes 2024-04-24T22:42:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1580583936 bytes 2024-04-24T22:42:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1626247168 bytes 2024-04-24T22:42:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1640463360 bytes 2024-04-24T22:42:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1686089728 bytes 2024-04-24T22:42:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1753601024 bytes 2024-04-24T22:42:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1764806656 bytes 2024-04-24T22:42:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1775848448 bytes 2024-04-24T22:42:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1838726144 bytes 2024-04-24T22:42:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1867729920 bytes 2024-04-24T22:42:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 1894508544 bytes 2024-04-24T22:42:19.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 1922787328 bytes 2024-04-24T22:42:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 1975161856 bytes 2024-04-24T22:42:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 1980823552 bytes 2024-04-24T22:42:20.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 1985135616 bytes 2024-04-24T22:42:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:42:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:42:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:42:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:42:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:42:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_20 2024-04-24T22:42:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2024-04-24T22:42:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T22:42:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:42:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:42:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_20 128M 2024-04-24T22:42:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T22:42:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:42:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:42:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:42:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_20 2024-04-24T22:42:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:42:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2024-04-24T22:42:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T22:42:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:42:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2024-04-24T22:42:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:42:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2024-04-24T22:42:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_20': 2024-04-24T22:42:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:42:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:42:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:42:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 135dd5e872ca 2024-04-24T22:42:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.135dd5e872ca 2024-04-24T22:42:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:42:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:42:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:42:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:42:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:42:20 2024 2024-04-24T22:42:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:42:20 2024 2024-04-24T22:42:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:42:20 2024 2024-04-24T22:42:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:42:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:42:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 061306d6-3c6a-4f11-84ad-c1042b24a00e 2024-04-24T22:42:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:42:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:42:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_20 2024-04-24T22:42:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T22:42:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:42:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=7 2024-04-24T22:42:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:42:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:42:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 14419968 bytes 2024-04-24T22:42:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 28469248 bytes 2024-04-24T22:42:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 29074432 bytes 2024-04-24T22:42:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 81617920 bytes 2024-04-24T22:42:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 104877056 bytes 2024-04-24T22:42:22.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 139195392 bytes 2024-04-24T22:42:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 182223872 bytes 2024-04-24T22:42:22.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 237793280 bytes 2024-04-24T22:42:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 272329728 bytes 2024-04-24T22:42:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 309391360 bytes 2024-04-24T22:42:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 371258368 bytes 2024-04-24T22:42:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 430586880 bytes 2024-04-24T22:42:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 446440448 bytes 2024-04-24T22:42:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 460445696 bytes 2024-04-24T22:42:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 461086720 bytes 2024-04-24T22:42:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 535951360 bytes 2024-04-24T22:42:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 571753472 bytes 2024-04-24T22:42:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 592802816 bytes 2024-04-24T22:42:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 617573376 bytes 2024-04-24T22:42:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 643128320 bytes 2024-04-24T22:42:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 667910144 bytes 2024-04-24T22:42:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 689928192 bytes 2024-04-24T22:42:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 728820736 bytes 2024-04-24T22:42:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 783462400 bytes 2024-04-24T22:42:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 802231296 bytes 2024-04-24T22:42:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 827229184 bytes 2024-04-24T22:42:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 882439168 bytes 2024-04-24T22:42:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 939091968 bytes 2024-04-24T22:42:25.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 974127104 bytes 2024-04-24T22:42:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1040047104 bytes 2024-04-24T22:42:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1087348736 bytes 2024-04-24T22:42:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1118154752 bytes 2024-04-24T22:42:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1174740992 bytes 2024-04-24T22:42:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1243161600 bytes 2024-04-24T22:42:26.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1272646656 bytes 2024-04-24T22:42:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1325618176 bytes 2024-04-24T22:42:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1385118720 bytes 2024-04-24T22:42:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1424082944 bytes 2024-04-24T22:42:26.619 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T22:42:26.614+0000 7fb1e9f85700 -1 reset not still connected to 0x5586bb0995f0 2024-04-24T22:42:26.619 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T22:42:26.617+0000 7fb1e9f85700 -1 reset not still connected to 0x5586bcb39ee0 2024-04-24T22:42:26.619 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T22:42:26.617+0000 7fb1e9f85700 -1 reset not still connected to 0x5586cbeaec30 2024-04-24T22:42:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1459268608 bytes 2024-04-24T22:42:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1512741888 bytes 2024-04-24T22:42:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1559915520 bytes 2024-04-24T22:42:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1571630080 bytes 2024-04-24T22:42:27.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1598638080 bytes 2024-04-24T22:42:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1640647680 bytes 2024-04-24T22:42:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1706504192 bytes 2024-04-24T22:42:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1716792320 bytes 2024-04-24T22:42:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1784782848 bytes 2024-04-24T22:42:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1852964864 bytes 2024-04-24T22:42:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 1877833728 bytes 2024-04-24T22:42:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 1944379392 bytes 2024-04-24T22:42:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:42:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:42:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:42:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:42:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:42:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_21 2024-04-24T22:42:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2024-04-24T22:42:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T22:42:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:42:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:42:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_21 128M 2024-04-24T22:42:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T22:42:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:42:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:42:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:42:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_21 2024-04-24T22:42:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:42:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2024-04-24T22:42:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T22:42:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:42:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2024-04-24T22:42:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:42:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2024-04-24T22:42:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_21': 2024-04-24T22:42:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:42:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:42:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:42:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 1369ca3834a9 2024-04-24T22:42:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.1369ca3834a9 2024-04-24T22:42:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:42:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:42:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:42:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:42:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:42:28 2024 2024-04-24T22:42:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:42:28 2024 2024-04-24T22:42:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:42:28 2024 2024-04-24T22:42:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:42:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:42:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 90338b6b-d62c-42ea-9b32-905a5d005824 2024-04-24T22:42:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:42:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:42:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_21 2024-04-24T22:42:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T22:42:29.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:42:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=11 2024-04-24T22:42:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:42:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:42:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 47832064 bytes 2024-04-24T22:42:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 48420864 bytes 2024-04-24T22:42:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 96330752 bytes 2024-04-24T22:42:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 99992576 bytes 2024-04-24T22:42:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 133511168 bytes 2024-04-24T22:42:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 169186304 bytes 2024-04-24T22:42:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 228429824 bytes 2024-04-24T22:42:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 229049344 bytes 2024-04-24T22:42:30.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 310094848 bytes 2024-04-24T22:42:30.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 363156480 bytes 2024-04-24T22:42:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 404951040 bytes 2024-04-24T22:42:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 467580928 bytes 2024-04-24T22:42:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 489653248 bytes 2024-04-24T22:42:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 558469120 bytes 2024-04-24T22:42:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 617276416 bytes 2024-04-24T22:42:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 618464256 bytes 2024-04-24T22:42:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 685193216 bytes 2024-04-24T22:42:31.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 771992576 bytes 2024-04-24T22:42:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 837474304 bytes 2024-04-24T22:42:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 905452544 bytes 2024-04-24T22:42:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 928761856 bytes 2024-04-24T22:42:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 947491840 bytes 2024-04-24T22:42:32.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 963643392 bytes 2024-04-24T22:42:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 993875968 bytes 2024-04-24T22:42:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1015710720 bytes 2024-04-24T22:42:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1056478208 bytes 2024-04-24T22:42:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1075043328 bytes 2024-04-24T22:42:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1104111616 bytes 2024-04-24T22:42:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1159651328 bytes 2024-04-24T22:42:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1188161536 bytes 2024-04-24T22:42:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1199375360 bytes 2024-04-24T22:42:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1203095552 bytes 2024-04-24T22:42:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1232188416 bytes 2024-04-24T22:42:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1291317248 bytes 2024-04-24T22:42:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1368663040 bytes 2024-04-24T22:42:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1402568704 bytes 2024-04-24T22:42:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1436525568 bytes 2024-04-24T22:42:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1481308160 bytes 2024-04-24T22:42:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1524630528 bytes 2024-04-24T22:42:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1562475520 bytes 2024-04-24T22:42:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1592790016 bytes 2024-04-24T22:42:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1605151744 bytes 2024-04-24T22:42:35.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1655053312 bytes 2024-04-24T22:42:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1712861184 bytes 2024-04-24T22:42:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1757651968 bytes 2024-04-24T22:42:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1830036480 bytes 2024-04-24T22:42:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1921019904 bytes 2024-04-24T22:42:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1990995968 bytes 2024-04-24T22:42:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2036204544 bytes 2024-04-24T22:42:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2036819968 bytes 2024-04-24T22:42:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2092505088 bytes 2024-04-24T22:42:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2185995264 bytes 2024-04-24T22:42:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2254830592 bytes 2024-04-24T22:42:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2294005760 bytes 2024-04-24T22:42:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2432752640 bytes 2024-04-24T22:42:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2460641280 bytes 2024-04-24T22:42:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2511216640 bytes 2024-04-24T22:42:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2511835136 bytes 2024-04-24T22:42:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2579567616 bytes 2024-04-24T22:42:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2651921408 bytes 2024-04-24T22:42:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2677928960 bytes 2024-04-24T22:42:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2711997440 bytes 2024-04-24T22:42:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2763526144 bytes 2024-04-24T22:42:38.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2803321856 bytes 2024-04-24T22:42:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2843149312 bytes 2024-04-24T22:42:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2855138304 bytes 2024-04-24T22:42:38.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2934697984 bytes 2024-04-24T22:42:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2972165120 bytes 2024-04-24T22:42:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 3009283072 bytes 2024-04-24T22:42:39.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 3026586624 bytes 2024-04-24T22:42:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 3071156224 bytes 2024-04-24T22:42:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 3115699200 bytes 2024-04-24T22:42:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 3146103808 bytes 2024-04-24T22:42:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 3249130496 bytes 2024-04-24T22:42:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 3295960064 bytes 2024-04-24T22:42:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 3301112832 bytes 2024-04-24T22:42:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 3371166720 bytes 2024-04-24T22:42:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:42:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:42:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:42:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:42:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:42:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_22 2024-04-24T22:42:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2024-04-24T22:42:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T22:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_22 128M 2024-04-24T22:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T22:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:42:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_22 2024-04-24T22:42:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:42:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2024-04-24T22:42:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T22:42:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:42:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2024-04-24T22:42:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:42:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2024-04-24T22:42:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_22': 2024-04-24T22:42:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:42:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:42:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:42:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 13753417351 2024-04-24T22:42:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.13753417351 2024-04-24T22:42:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:42:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:42:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:42:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:42:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:42:40 2024 2024-04-24T22:42:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:42:40 2024 2024-04-24T22:42:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:42:40 2024 2024-04-24T22:42:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:42:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:42:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 6ab5e540-e8f6-406b-9bdd-00cf740c3964 2024-04-24T22:42:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:42:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:42:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_22 2024-04-24T22:42:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T22:42:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:42:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=6 2024-04-24T22:42:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:42:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:42:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 31120384 bytes 2024-04-24T22:42:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 72574976 bytes 2024-04-24T22:42:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 134775808 bytes 2024-04-24T22:42:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 164913152 bytes 2024-04-24T22:42:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 209596416 bytes 2024-04-24T22:42:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 318319616 bytes 2024-04-24T22:42:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 441364480 bytes 2024-04-24T22:42:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 464819200 bytes 2024-04-24T22:42:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 526545920 bytes 2024-04-24T22:42:42.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 598901760 bytes 2024-04-24T22:42:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 610096128 bytes 2024-04-24T22:42:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 664902656 bytes 2024-04-24T22:42:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 692353024 bytes 2024-04-24T22:42:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 696073216 bytes 2024-04-24T22:42:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 731876352 bytes 2024-04-24T22:42:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 805906432 bytes 2024-04-24T22:42:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 830649344 bytes 2024-04-24T22:42:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 889151488 bytes 2024-04-24T22:42:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 922722304 bytes 2024-04-24T22:42:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 1006284800 bytes 2024-04-24T22:42:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 1016623104 bytes 2024-04-24T22:42:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 1095367680 bytes 2024-04-24T22:42:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1155367936 bytes 2024-04-24T22:42:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1228537856 bytes 2024-04-24T22:42:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1287168000 bytes 2024-04-24T22:42:45.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1330049024 bytes 2024-04-24T22:42:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1353710592 bytes 2024-04-24T22:42:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1369788416 bytes 2024-04-24T22:42:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1382422528 bytes 2024-04-24T22:42:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1414511616 bytes 2024-04-24T22:42:45.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1455133696 bytes 2024-04-24T22:42:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1492415488 bytes 2024-04-24T22:42:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1539648512 bytes 2024-04-24T22:42:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1635419136 bytes 2024-04-24T22:42:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1709303808 bytes 2024-04-24T22:42:46.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1762360320 bytes 2024-04-24T22:42:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1786632192 bytes 2024-04-24T22:42:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1826161664 bytes 2024-04-24T22:42:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1921560576 bytes 2024-04-24T22:42:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 2009457664 bytes 2024-04-24T22:42:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:42:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:42:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:42:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:42:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:42:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_23 2024-04-24T22:42:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2024-04-24T22:42:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T22:42:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:42:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:42:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_23 128M 2024-04-24T22:42:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T22:42:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:42:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:42:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:42:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_23 2024-04-24T22:42:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:42:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2024-04-24T22:42:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T22:42:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:42:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2024-04-24T22:42:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:42:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2024-04-24T22:42:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_23': 2024-04-24T22:42:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:42:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:42:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:42:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 13819a483112 2024-04-24T22:42:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.13819a483112 2024-04-24T22:42:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:42:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:42:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:42:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:42:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:42:47 2024 2024-04-24T22:42:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:42:47 2024 2024-04-24T22:42:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:42:47 2024 2024-04-24T22:42:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:42:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:42:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: c24b92fa-65bf-4177-9441-bcf5f91f92ff 2024-04-24T22:42:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:42:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:42:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_23 2024-04-24T22:42:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T22:42:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:42:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=9 2024-04-24T22:42:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:42:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:42:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 28368896 bytes 2024-04-24T22:42:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 86844416 bytes 2024-04-24T22:42:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 92645376 bytes 2024-04-24T22:42:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 155064320 bytes 2024-04-24T22:42:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 186560512 bytes 2024-04-24T22:42:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 237807616 bytes 2024-04-24T22:42:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 333881344 bytes 2024-04-24T22:42:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 360923136 bytes 2024-04-24T22:42:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 426735616 bytes 2024-04-24T22:42:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 500152320 bytes 2024-04-24T22:42:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 538805248 bytes 2024-04-24T22:42:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 571825152 bytes 2024-04-24T22:42:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 608634880 bytes 2024-04-24T22:42:50.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 655144960 bytes 2024-04-24T22:42:50.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 730901504 bytes 2024-04-24T22:42:50.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 778430464 bytes 2024-04-24T22:42:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 812511232 bytes 2024-04-24T22:42:50.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 829868032 bytes 2024-04-24T22:42:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 907067392 bytes 2024-04-24T22:42:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 981558272 bytes 2024-04-24T22:42:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 1027578880 bytes 2024-04-24T22:42:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 1058476032 bytes 2024-04-24T22:42:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1142523904 bytes 2024-04-24T22:42:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1195874304 bytes 2024-04-24T22:42:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1211438080 bytes 2024-04-24T22:42:52.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1224603648 bytes 2024-04-24T22:42:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1266226176 bytes 2024-04-24T22:42:52.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1344839680 bytes 2024-04-24T22:42:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1384824832 bytes 2024-04-24T22:42:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1439062016 bytes 2024-04-24T22:42:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1458843648 bytes 2024-04-24T22:42:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1539416064 bytes 2024-04-24T22:42:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1584766976 bytes 2024-04-24T22:42:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1650570240 bytes 2024-04-24T22:42:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1701943296 bytes 2024-04-24T22:42:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1710517248 bytes 2024-04-24T22:42:53.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1788550144 bytes 2024-04-24T22:42:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1824320512 bytes 2024-04-24T22:42:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1846070272 bytes 2024-04-24T22:42:54.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1859898368 bytes 2024-04-24T22:42:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1939984384 bytes 2024-04-24T22:42:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1991171072 bytes 2024-04-24T22:42:54.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 2055607296 bytes 2024-04-24T22:42:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 2133804032 bytes 2024-04-24T22:42:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 2161280000 bytes 2024-04-24T22:42:55.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 2211121152 bytes 2024-04-24T22:42:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 2215312384 bytes 2024-04-24T22:42:55.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2236232704 bytes 2024-04-24T22:42:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2264223744 bytes 2024-04-24T22:42:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2299252736 bytes 2024-04-24T22:42:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2391354368 bytes 2024-04-24T22:42:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2428546048 bytes 2024-04-24T22:42:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2484255744 bytes 2024-04-24T22:42:56.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2562009088 bytes 2024-04-24T22:42:56.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2644719616 bytes 2024-04-24T22:42:56.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2709862400 bytes 2024-04-24T22:42:56.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2780645376 bytes 2024-04-24T22:42:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2795292672 bytes 2024-04-24T22:42:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2843722752 bytes 2024-04-24T22:42:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:42:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:42:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:42:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:42:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:42:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_24 2024-04-24T22:42:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2024-04-24T22:42:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T22:42:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:42:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:42:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_24 128M 2024-04-24T22:42:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T22:42:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:42:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:42:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:42:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:42:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_24 2024-04-24T22:42:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:42:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2024-04-24T22:42:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T22:42:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:42:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2024-04-24T22:42:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:42:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2024-04-24T22:42:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_24': 2024-04-24T22:42:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:42:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:42:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:42:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 138da78696e6 2024-04-24T22:42:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.138da78696e6 2024-04-24T22:42:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:42:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:42:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:42:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:42:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:42:57 2024 2024-04-24T22:42:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:42:57 2024 2024-04-24T22:42:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:42:57 2024 2024-04-24T22:42:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:42:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:42:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: f25eb5c2-2f4d-4c6e-9c1e-1ee69ca3048a 2024-04-24T22:42:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:42:58.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:42:58.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_24 2024-04-24T22:42:58.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:42:58.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:42:58.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T22:42:58.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:42:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=12 2024-04-24T22:42:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:42:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:42:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 26567680 bytes 2024-04-24T22:42:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 149945344 bytes 2024-04-24T22:42:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 188957696 bytes 2024-04-24T22:42:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 224234496 bytes 2024-04-24T22:42:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 248665088 bytes 2024-04-24T22:42:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 249283584 bytes 2024-04-24T22:42:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 306683904 bytes 2024-04-24T22:42:59.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 325310464 bytes 2024-04-24T22:42:59.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 359827456 bytes 2024-04-24T22:42:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 387731456 bytes 2024-04-24T22:42:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 447152128 bytes 2024-04-24T22:42:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 477266944 bytes 2024-04-24T22:43:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 516652032 bytes 2024-04-24T22:43:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 584565760 bytes 2024-04-24T22:43:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 653415424 bytes 2024-04-24T22:43:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 686166016 bytes 2024-04-24T22:43:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 773583872 bytes 2024-04-24T22:43:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 782193664 bytes 2024-04-24T22:43:00.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 821358592 bytes 2024-04-24T22:43:01.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 856518656 bytes 2024-04-24T22:43:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 911139840 bytes 2024-04-24T22:43:01.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 936507392 bytes 2024-04-24T22:43:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 978564096 bytes 2024-04-24T22:43:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 994130944 bytes 2024-04-24T22:43:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1053064192 bytes 2024-04-24T22:43:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1080579072 bytes 2024-04-24T22:43:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1143824384 bytes 2024-04-24T22:43:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1192881152 bytes 2024-04-24T22:43:02.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1210583040 bytes 2024-04-24T22:43:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1221295104 bytes 2024-04-24T22:43:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1295482880 bytes 2024-04-24T22:43:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1323136000 bytes 2024-04-24T22:43:03.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1349411840 bytes 2024-04-24T22:43:03.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1368666112 bytes 2024-04-24T22:43:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1413104640 bytes 2024-04-24T22:43:03.406 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-24T22:43:03.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-24T22:43:03.406 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-04-24T22:43:03.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:43:03.404+0000 7fc075838700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 58494) UID: 1000 2024-04-24T22:43:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1444579328 bytes 2024-04-24T22:43:03.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:43:03.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1471915008 bytes 2024-04-24T22:43:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1543022592 bytes 2024-04-24T22:43:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1589307392 bytes 2024-04-24T22:43:04.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1608990720 bytes 2024-04-24T22:43:04.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1649344512 bytes 2024-04-24T22:43:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1704107008 bytes 2024-04-24T22:43:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1744197632 bytes 2024-04-24T22:43:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1801299968 bytes 2024-04-24T22:43:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1865707520 bytes 2024-04-24T22:43:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1878737920 bytes 2024-04-24T22:43:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1938484224 bytes 2024-04-24T22:43:05.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1967088640 bytes 2024-04-24T22:43:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 1985038336 bytes 2024-04-24T22:43:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2025083904 bytes 2024-04-24T22:43:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2077325312 bytes 2024-04-24T22:43:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2104331264 bytes 2024-04-24T22:43:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2126539776 bytes 2024-04-24T22:43:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2216953856 bytes 2024-04-24T22:43:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2250727424 bytes 2024-04-24T22:43:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2289436672 bytes 2024-04-24T22:43:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2312244224 bytes 2024-04-24T22:43:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2334753792 bytes 2024-04-24T22:43:06.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2335342592 bytes 2024-04-24T22:43:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2335955968 bytes 2024-04-24T22:43:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2430949376 bytes 2024-04-24T22:43:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2470768640 bytes 2024-04-24T22:43:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2521072640 bytes 2024-04-24T22:43:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2614180864 bytes 2024-04-24T22:43:07.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2676517888 bytes 2024-04-24T22:43:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2716106752 bytes 2024-04-24T22:43:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2783637504 bytes 2024-04-24T22:43:07.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2801821696 bytes 2024-04-24T22:43:08.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2875423744 bytes 2024-04-24T22:43:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 2902476800 bytes 2024-04-24T22:43:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 2947426304 bytes 2024-04-24T22:43:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 2948042752 bytes 2024-04-24T22:43:08.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 2970557440 bytes 2024-04-24T22:43:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 2997073920 bytes 2024-04-24T22:43:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 3068526592 bytes 2024-04-24T22:43:09.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 3126797312 bytes 2024-04-24T22:43:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 3144960000 bytes 2024-04-24T22:43:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 3166061568 bytes 2024-04-24T22:43:09.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7900 IOs, 3242452992 bytes 2024-04-24T22:43:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8000 IOs, 3259994112 bytes 2024-04-24T22:43:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8100 IOs, 3321383936 bytes 2024-04-24T22:43:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8200 IOs, 3346483200 bytes 2024-04-24T22:43:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8300 IOs, 3395211264 bytes 2024-04-24T22:43:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8400 IOs, 3417907200 bytes 2024-04-24T22:43:10.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:43:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:43:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:43:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:43:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:43:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_25 2024-04-24T22:43:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2024-04-24T22:43:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:43:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:43:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T22:43:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:43:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:43:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_25 128M 2024-04-24T22:43:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:43:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:43:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T22:43:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:43:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:43:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:43:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_25 2024-04-24T22:43:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:43:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2024-04-24T22:43:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:43:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:43:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T22:43:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:43:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2024-04-24T22:43:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:43:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2024-04-24T22:43:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_25': 2024-04-24T22:43:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:43:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:43:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:43:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 13996384dcad 2024-04-24T22:43:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.13996384dcad 2024-04-24T22:43:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:43:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:43:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:43:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:43:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:43:10 2024 2024-04-24T22:43:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:43:10 2024 2024-04-24T22:43:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:43:10 2024 2024-04-24T22:43:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:43:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:43:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: a6830866-682c-4c96-8b3b-75517871f35d 2024-04-24T22:43:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:43:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:43:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_25 2024-04-24T22:43:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:43:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:43:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T22:43:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:43:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=14 2024-04-24T22:43:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:43:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:43:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 11438080 bytes 2024-04-24T22:43:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 52065280 bytes 2024-04-24T22:43:11.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 71725056 bytes 2024-04-24T22:43:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 156161024 bytes 2024-04-24T22:43:11.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 211165184 bytes 2024-04-24T22:43:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 219410432 bytes 2024-04-24T22:43:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 275325952 bytes 2024-04-24T22:43:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 299726848 bytes 2024-04-24T22:43:12.407 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-24T22:43:12.408 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:43:12.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:43:12.408 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-24T22:43:12.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-24T22:43:12.408 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-24T22:43:12.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-24T22:43:12.411 INFO:tasks.rbd_mirror_thrash:waiting for 104 secs before thrashing 2024-04-24T22:43:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 400109568 bytes 2024-04-24T22:43:12.629 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-24T22:43:12.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-24T22:43:12.630 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-04-24T22:43:12.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T22:43:12.630+0000 7fc33f2da700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 54203) UID: 1000 2024-04-24T22:43:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 400732160 bytes 2024-04-24T22:43:12.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:43:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 430748672 bytes 2024-04-24T22:43:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 488741888 bytes 2024-04-24T22:43:13.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 515862528 bytes 2024-04-24T22:43:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 543862784 bytes 2024-04-24T22:43:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 559653888 bytes 2024-04-24T22:43:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 607902720 bytes 2024-04-24T22:43:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 608516096 bytes 2024-04-24T22:43:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 661054464 bytes 2024-04-24T22:43:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 745296896 bytes 2024-04-24T22:43:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 788822016 bytes 2024-04-24T22:43:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 863709184 bytes 2024-04-24T22:43:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 960525312 bytes 2024-04-24T22:43:14.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 982131712 bytes 2024-04-24T22:43:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1013872640 bytes 2024-04-24T22:43:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1053513728 bytes 2024-04-24T22:43:15.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1124959232 bytes 2024-04-24T22:43:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1163323392 bytes 2024-04-24T22:43:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1181498368 bytes 2024-04-24T22:43:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1199016960 bytes 2024-04-24T22:43:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1215212544 bytes 2024-04-24T22:43:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1235192832 bytes 2024-04-24T22:43:15.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1320619008 bytes 2024-04-24T22:43:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1337833472 bytes 2024-04-24T22:43:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1374466048 bytes 2024-04-24T22:43:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1439337472 bytes 2024-04-24T22:43:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1484789760 bytes 2024-04-24T22:43:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1552038912 bytes 2024-04-24T22:43:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1587639296 bytes 2024-04-24T22:43:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1662644224 bytes 2024-04-24T22:43:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1669343232 bytes 2024-04-24T22:43:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1714231296 bytes 2024-04-24T22:43:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1729282048 bytes 2024-04-24T22:43:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1748164608 bytes 2024-04-24T22:43:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1787713536 bytes 2024-04-24T22:43:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1850570752 bytes 2024-04-24T22:43:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1873327104 bytes 2024-04-24T22:43:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1886472192 bytes 2024-04-24T22:43:18.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1926518784 bytes 2024-04-24T22:43:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 1927129088 bytes 2024-04-24T22:43:18.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 1996046336 bytes 2024-04-24T22:43:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2007347200 bytes 2024-04-24T22:43:18.631 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-24T22:43:18.631 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:43:18.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:43:18.632 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-24T22:43:18.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-24T22:43:18.632 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-24T22:43:18.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-24T22:43:18.636 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2024-04-24T22:43:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2124271616 bytes 2024-04-24T22:43:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2207375360 bytes 2024-04-24T22:43:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2265326592 bytes 2024-04-24T22:43:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2284880896 bytes 2024-04-24T22:43:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2312881152 bytes 2024-04-24T22:43:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2355501056 bytes 2024-04-24T22:43:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2386053120 bytes 2024-04-24T22:43:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2423162880 bytes 2024-04-24T22:43:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2497773568 bytes 2024-04-24T22:43:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2500417536 bytes 2024-04-24T22:43:20.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2560648192 bytes 2024-04-24T22:43:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2633270272 bytes 2024-04-24T22:43:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2714475520 bytes 2024-04-24T22:43:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2719046656 bytes 2024-04-24T22:43:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2739939328 bytes 2024-04-24T22:43:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2795290624 bytes 2024-04-24T22:43:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2858663936 bytes 2024-04-24T22:43:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2901730304 bytes 2024-04-24T22:43:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 2977106944 bytes 2024-04-24T22:43:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 3012883456 bytes 2024-04-24T22:43:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 3052036096 bytes 2024-04-24T22:43:21.670 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:43:21.668+0000 7f0f4ef7b700 -1 reset not still connected to 0x561388b129c0 2024-04-24T22:43:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 3108101120 bytes 2024-04-24T22:43:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 3154620416 bytes 2024-04-24T22:43:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 3202264064 bytes 2024-04-24T22:43:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 3228026880 bytes 2024-04-24T22:43:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 3264348160 bytes 2024-04-24T22:43:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 3339393024 bytes 2024-04-24T22:43:22.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7900 IOs, 3364147200 bytes 2024-04-24T22:43:22.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8000 IOs, 3403560960 bytes 2024-04-24T22:43:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8100 IOs, 3478200320 bytes 2024-04-24T22:43:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8200 IOs, 3577035776 bytes 2024-04-24T22:43:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8300 IOs, 3632583680 bytes 2024-04-24T22:43:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8400 IOs, 3655327744 bytes 2024-04-24T22:43:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8500 IOs, 3723734016 bytes 2024-04-24T22:43:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8600 IOs, 3751417856 bytes 2024-04-24T22:43:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8700 IOs, 3816392704 bytes 2024-04-24T22:43:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8800 IOs, 3872603136 bytes 2024-04-24T22:43:24.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8900 IOs, 4070047744 bytes 2024-04-24T22:43:24.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9000 IOs, 4127331328 bytes 2024-04-24T22:43:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9100 IOs, 4149185536 bytes 2024-04-24T22:43:24.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9200 IOs, 4200624128 bytes 2024-04-24T22:43:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9300 IOs, 4232153088 bytes 2024-04-24T22:43:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9400 IOs, 4260359168 bytes 2024-04-24T22:43:25.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9500 IOs, 4320000000 bytes 2024-04-24T22:43:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:43:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:43:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:43:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:43:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:43:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_26 2024-04-24T22:43:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2024-04-24T22:43:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:43:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:43:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T22:43:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:43:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:43:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_26 128M 2024-04-24T22:43:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:43:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:43:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T22:43:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:43:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:43:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:43:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_26 2024-04-24T22:43:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:43:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2024-04-24T22:43:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:43:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:43:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T22:43:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:43:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2024-04-24T22:43:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:43:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2024-04-24T22:43:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_26': 2024-04-24T22:43:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:43:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:43:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:43:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 13b3b9bf26d1 2024-04-24T22:43:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.13b3b9bf26d1 2024-04-24T22:43:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:43:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:43:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:43:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:43:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:43:25 2024 2024-04-24T22:43:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:43:25 2024 2024-04-24T22:43:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:43:25 2024 2024-04-24T22:43:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:43:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:43:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: f38aef72-1054-47c3-917d-19c1b6627e53 2024-04-24T22:43:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:43:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:43:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_26 2024-04-24T22:43:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:43:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:43:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T22:43:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:43:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=13 2024-04-24T22:43:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:43:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:43:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 47792128 bytes 2024-04-24T22:43:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 161139712 bytes 2024-04-24T22:43:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 199848960 bytes 2024-04-24T22:43:26.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 279390208 bytes 2024-04-24T22:43:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 300485632 bytes 2024-04-24T22:43:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 363516928 bytes 2024-04-24T22:43:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 421480448 bytes 2024-04-24T22:43:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 478585856 bytes 2024-04-24T22:43:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 577148928 bytes 2024-04-24T22:43:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 598348800 bytes 2024-04-24T22:43:28.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 712120320 bytes 2024-04-24T22:43:28.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 736555008 bytes 2024-04-24T22:43:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 752053248 bytes 2024-04-24T22:43:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 845845504 bytes 2024-04-24T22:43:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 922241024 bytes 2024-04-24T22:43:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 947570688 bytes 2024-04-24T22:43:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 967109632 bytes 2024-04-24T22:43:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 1048015872 bytes 2024-04-24T22:43:29.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 1079352320 bytes 2024-04-24T22:43:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 1094263808 bytes 2024-04-24T22:43:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 1113341952 bytes 2024-04-24T22:43:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 1113954304 bytes 2024-04-24T22:43:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1142685696 bytes 2024-04-24T22:43:29.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1212398592 bytes 2024-04-24T22:43:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1255971840 bytes 2024-04-24T22:43:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1317440512 bytes 2024-04-24T22:43:30.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1365370880 bytes 2024-04-24T22:43:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1390417920 bytes 2024-04-24T22:43:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1417939968 bytes 2024-04-24T22:43:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1463772160 bytes 2024-04-24T22:43:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1487371264 bytes 2024-04-24T22:43:30.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1512514560 bytes 2024-04-24T22:43:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1556048896 bytes 2024-04-24T22:43:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1588044800 bytes 2024-04-24T22:43:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1602264064 bytes 2024-04-24T22:43:31.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1604668416 bytes 2024-04-24T22:43:31.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1663715328 bytes 2024-04-24T22:43:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1741890560 bytes 2024-04-24T22:43:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1784634368 bytes 2024-04-24T22:43:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1799274496 bytes 2024-04-24T22:43:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1820012544 bytes 2024-04-24T22:43:32.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1860078592 bytes 2024-04-24T22:43:32.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1870088192 bytes 2024-04-24T22:43:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1900497920 bytes 2024-04-24T22:43:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1942361088 bytes 2024-04-24T22:43:32.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 2020400128 bytes 2024-04-24T22:43:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 2050612224 bytes 2024-04-24T22:43:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2118823936 bytes 2024-04-24T22:43:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2205200384 bytes 2024-04-24T22:43:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2224655360 bytes 2024-04-24T22:43:33.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2283320320 bytes 2024-04-24T22:43:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2310800384 bytes 2024-04-24T22:43:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2363066368 bytes 2024-04-24T22:43:34.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2411572224 bytes 2024-04-24T22:43:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2417526784 bytes 2024-04-24T22:43:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2481299456 bytes 2024-04-24T22:43:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2516285440 bytes 2024-04-24T22:43:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2545334272 bytes 2024-04-24T22:43:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2555923456 bytes 2024-04-24T22:43:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2596365312 bytes 2024-04-24T22:43:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2613868544 bytes 2024-04-24T22:43:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2712672256 bytes 2024-04-24T22:43:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2785138688 bytes 2024-04-24T22:43:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2832243712 bytes 2024-04-24T22:43:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2919392256 bytes 2024-04-24T22:43:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 3003224064 bytes 2024-04-24T22:43:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 3035014144 bytes 2024-04-24T22:43:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 3071097856 bytes 2024-04-24T22:43:36.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 3150523392 bytes 2024-04-24T22:43:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 3208819712 bytes 2024-04-24T22:43:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 3227107328 bytes 2024-04-24T22:43:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 3278467072 bytes 2024-04-24T22:43:36.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 3371776000 bytes 2024-04-24T22:43:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 3393256448 bytes 2024-04-24T22:43:37.094 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 3414921216 bytes 2024-04-24T22:43:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 3443573760 bytes 2024-04-24T22:43:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 3460549632 bytes 2024-04-24T22:43:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 3493045248 bytes 2024-04-24T22:43:37.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7900 IOs, 3544000512 bytes 2024-04-24T22:43:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8000 IOs, 3618338816 bytes 2024-04-24T22:43:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8100 IOs, 3621893120 bytes 2024-04-24T22:43:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8200 IOs, 3649984512 bytes 2024-04-24T22:43:38.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8300 IOs, 3699532800 bytes 2024-04-24T22:43:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8400 IOs, 3720927232 bytes 2024-04-24T22:43:38.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8500 IOs, 3755038720 bytes 2024-04-24T22:43:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8600 IOs, 3827369984 bytes 2024-04-24T22:43:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8700 IOs, 3847159808 bytes 2024-04-24T22:43:38.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8800 IOs, 3847749632 bytes 2024-04-24T22:43:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8900 IOs, 3873955840 bytes 2024-04-24T22:43:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9000 IOs, 3906556928 bytes 2024-04-24T22:43:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9100 IOs, 3929093120 bytes 2024-04-24T22:43:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:43:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:43:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:43:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:43:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:43:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_27 2024-04-24T22:43:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2024-04-24T22:43:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:43:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:43:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T22:43:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:43:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:43:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_27 128M 2024-04-24T22:43:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:43:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:43:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T22:43:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:43:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:43:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:43:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_27 2024-04-24T22:43:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:43:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2024-04-24T22:43:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:43:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:43:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T22:43:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:43:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2024-04-24T22:43:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:43:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2024-04-24T22:43:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_27': 2024-04-24T22:43:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:43:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:43:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:43:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 13bfe8c5df48 2024-04-24T22:43:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.13bfe8c5df48 2024-04-24T22:43:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:43:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:43:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:43:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:43:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:43:39 2024 2024-04-24T22:43:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:43:39 2024 2024-04-24T22:43:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:43:39 2024 2024-04-24T22:43:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:43:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:43:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 1143f5a5-897d-4fc6-98d2-c436f71170b7 2024-04-24T22:43:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:43:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:43:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_27 2024-04-24T22:43:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:43:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:43:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T22:43:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:43:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=11 2024-04-24T22:43:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:43:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:43:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 50036736 bytes 2024-04-24T22:43:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 80369664 bytes 2024-04-24T22:43:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 107428864 bytes 2024-04-24T22:43:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 117839872 bytes 2024-04-24T22:43:40.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 132490240 bytes 2024-04-24T22:43:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 187697152 bytes 2024-04-24T22:43:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 200335360 bytes 2024-04-24T22:43:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 272110592 bytes 2024-04-24T22:43:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 274578432 bytes 2024-04-24T22:43:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 319170560 bytes 2024-04-24T22:43:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 319792128 bytes 2024-04-24T22:43:41.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 362517504 bytes 2024-04-24T22:43:41.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 363895808 bytes 2024-04-24T22:43:42.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 397766656 bytes 2024-04-24T22:43:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 435103744 bytes 2024-04-24T22:43:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 451773440 bytes 2024-04-24T22:43:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 481187840 bytes 2024-04-24T22:43:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 494467072 bytes 2024-04-24T22:43:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 569816064 bytes 2024-04-24T22:43:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 605945856 bytes 2024-04-24T22:43:43.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 634112000 bytes 2024-04-24T22:43:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 647743488 bytes 2024-04-24T22:43:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 691281920 bytes 2024-04-24T22:43:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 771388416 bytes 2024-04-24T22:43:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 826664960 bytes 2024-04-24T22:43:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 928253952 bytes 2024-04-24T22:43:43.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 947616768 bytes 2024-04-24T22:43:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 995347456 bytes 2024-04-24T22:43:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1099274240 bytes 2024-04-24T22:43:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1110360064 bytes 2024-04-24T22:43:44.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1162334208 bytes 2024-04-24T22:43:44.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1178459136 bytes 2024-04-24T22:43:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1246794752 bytes 2024-04-24T22:43:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1282022400 bytes 2024-04-24T22:43:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1357063168 bytes 2024-04-24T22:43:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1395665920 bytes 2024-04-24T22:43:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1409953792 bytes 2024-04-24T22:43:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1506061312 bytes 2024-04-24T22:43:45.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1576666112 bytes 2024-04-24T22:43:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1608141824 bytes 2024-04-24T22:43:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1653811200 bytes 2024-04-24T22:43:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1677982720 bytes 2024-04-24T22:43:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1712049152 bytes 2024-04-24T22:43:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1775931392 bytes 2024-04-24T22:43:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1808501760 bytes 2024-04-24T22:43:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1812852736 bytes 2024-04-24T22:43:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1859180544 bytes 2024-04-24T22:43:46.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1876516864 bytes 2024-04-24T22:43:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 1936428032 bytes 2024-04-24T22:43:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2003033088 bytes 2024-04-24T22:43:47.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2072442880 bytes 2024-04-24T22:43:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2092340224 bytes 2024-04-24T22:43:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2122499072 bytes 2024-04-24T22:43:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2167754752 bytes 2024-04-24T22:43:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2180521984 bytes 2024-04-24T22:43:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2269908992 bytes 2024-04-24T22:43:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2356852736 bytes 2024-04-24T22:43:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2379547648 bytes 2024-04-24T22:43:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2405532672 bytes 2024-04-24T22:43:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2489338880 bytes 2024-04-24T22:43:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2529102848 bytes 2024-04-24T22:43:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2531093504 bytes 2024-04-24T22:43:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2601935872 bytes 2024-04-24T22:43:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2632429568 bytes 2024-04-24T22:43:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2673729536 bytes 2024-04-24T22:43:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2710914048 bytes 2024-04-24T22:43:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2765746176 bytes 2024-04-24T22:43:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2795257856 bytes 2024-04-24T22:43:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2798571520 bytes 2024-04-24T22:43:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 2804559872 bytes 2024-04-24T22:43:50.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 2885103616 bytes 2024-04-24T22:43:50.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 2945073152 bytes 2024-04-24T22:43:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 2973254656 bytes 2024-04-24T22:43:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 3003944960 bytes 2024-04-24T22:43:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 3042700288 bytes 2024-04-24T22:43:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 3070662656 bytes 2024-04-24T22:43:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 3099346944 bytes 2024-04-24T22:43:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 3158948864 bytes 2024-04-24T22:43:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:43:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:43:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:43:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:43:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:43:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_28 2024-04-24T22:43:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2024-04-24T22:43:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:43:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:43:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T22:43:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:43:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:43:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_28 128M 2024-04-24T22:43:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:43:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:43:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T22:43:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:43:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:43:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:43:51.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:51.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_28 2024-04-24T22:43:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:43:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2024-04-24T22:43:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:43:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:43:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T22:43:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:43:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2024-04-24T22:43:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:43:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2024-04-24T22:43:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_28': 2024-04-24T22:43:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:43:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:43:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:43:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 13cba09d4eed 2024-04-24T22:43:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.13cba09d4eed 2024-04-24T22:43:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:43:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:43:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:43:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:43:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:43:51 2024 2024-04-24T22:43:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:43:51 2024 2024-04-24T22:43:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:43:51 2024 2024-04-24T22:43:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:43:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:43:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 0f1733b8-4c9d-4886-af25-8d2e8af19dd6 2024-04-24T22:43:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:43:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:43:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_28 2024-04-24T22:43:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:43:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:43:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T22:43:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:43:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=5 2024-04-24T22:43:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:43:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:43:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 2191360 bytes 2024-04-24T22:43:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 20812800 bytes 2024-04-24T22:43:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 83438592 bytes 2024-04-24T22:43:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 99979264 bytes 2024-04-24T22:43:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 121211904 bytes 2024-04-24T22:43:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 151460864 bytes 2024-04-24T22:43:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 152086528 bytes 2024-04-24T22:43:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 169678848 bytes 2024-04-24T22:43:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 187458560 bytes 2024-04-24T22:43:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 245899264 bytes 2024-04-24T22:43:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 318456832 bytes 2024-04-24T22:43:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 333104128 bytes 2024-04-24T22:43:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 335004672 bytes 2024-04-24T22:43:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 362262528 bytes 2024-04-24T22:43:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 365937664 bytes 2024-04-24T22:43:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 398609408 bytes 2024-04-24T22:43:54.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 410364928 bytes 2024-04-24T22:43:54.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 456825856 bytes 2024-04-24T22:43:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 491525120 bytes 2024-04-24T22:43:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 561693696 bytes 2024-04-24T22:43:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 602187776 bytes 2024-04-24T22:43:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 632153088 bytes 2024-04-24T22:43:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 653706240 bytes 2024-04-24T22:43:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 681254912 bytes 2024-04-24T22:43:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 695181312 bytes 2024-04-24T22:43:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 761409536 bytes 2024-04-24T22:43:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 848406528 bytes 2024-04-24T22:43:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 866177024 bytes 2024-04-24T22:43:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 898803712 bytes 2024-04-24T22:43:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 899438592 bytes 2024-04-24T22:43:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 955448320 bytes 2024-04-24T22:43:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1037067264 bytes 2024-04-24T22:43:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1109016576 bytes 2024-04-24T22:43:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1109647360 bytes 2024-04-24T22:43:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1172498432 bytes 2024-04-24T22:43:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1276056576 bytes 2024-04-24T22:43:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:43:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:43:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:43:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:43:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:43:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_29 2024-04-24T22:43:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2024-04-24T22:43:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:43:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:43:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T22:43:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:43:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:43:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_29 128M 2024-04-24T22:43:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:43:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:43:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T22:43:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:43:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:43:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:43:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:43:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_29 2024-04-24T22:43:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:43:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2024-04-24T22:43:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:43:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:43:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T22:43:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:43:57.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2024-04-24T22:43:58.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:43:58.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2024-04-24T22:43:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_29': 2024-04-24T22:43:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:43:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:43:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:43:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 13d7a30d6946 2024-04-24T22:43:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.13d7a30d6946 2024-04-24T22:43:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:43:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:43:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:43:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:43:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:43:57 2024 2024-04-24T22:43:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:43:57 2024 2024-04-24T22:43:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:43:57 2024 2024-04-24T22:43:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:43:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:43:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 171b9277-b7b3-4a7f-ae2b-c3d8aca7eadf 2024-04-24T22:43:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:43:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:43:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_29 2024-04-24T22:43:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:43:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:43:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T22:43:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:43:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=12 2024-04-24T22:43:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:43:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:43:58.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 96060416 bytes 2024-04-24T22:43:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 137907200 bytes 2024-04-24T22:43:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 218241024 bytes 2024-04-24T22:43:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 251840512 bytes 2024-04-24T22:43:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 309147648 bytes 2024-04-24T22:43:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 385342464 bytes 2024-04-24T22:43:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 416577536 bytes 2024-04-24T22:43:59.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 449084416 bytes 2024-04-24T22:43:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 491684864 bytes 2024-04-24T22:43:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 492326912 bytes 2024-04-24T22:43:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 519940096 bytes 2024-04-24T22:44:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 554211328 bytes 2024-04-24T22:44:00.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 608878592 bytes 2024-04-24T22:44:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 651963392 bytes 2024-04-24T22:44:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 721954816 bytes 2024-04-24T22:44:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 759931904 bytes 2024-04-24T22:44:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 805595136 bytes 2024-04-24T22:44:01.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 893348864 bytes 2024-04-24T22:44:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 927361024 bytes 2024-04-24T22:44:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 969060352 bytes 2024-04-24T22:44:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 1051565056 bytes 2024-04-24T22:44:01.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 1084539904 bytes 2024-04-24T22:44:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1100779520 bytes 2024-04-24T22:44:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1125790720 bytes 2024-04-24T22:44:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1157947392 bytes 2024-04-24T22:44:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1203819520 bytes 2024-04-24T22:44:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1309290496 bytes 2024-04-24T22:44:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1336553472 bytes 2024-04-24T22:44:02.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1364638720 bytes 2024-04-24T22:44:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1422050304 bytes 2024-04-24T22:44:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1469185024 bytes 2024-04-24T22:44:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1500019712 bytes 2024-04-24T22:44:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1531378688 bytes 2024-04-24T22:44:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1595036672 bytes 2024-04-24T22:44:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1673566208 bytes 2024-04-24T22:44:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1720974336 bytes 2024-04-24T22:44:04.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1779579904 bytes 2024-04-24T22:44:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1830363136 bytes 2024-04-24T22:44:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1874659328 bytes 2024-04-24T22:44:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1911532544 bytes 2024-04-24T22:44:04.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1916312576 bytes 2024-04-24T22:44:04.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1974815744 bytes 2024-04-24T22:44:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1985301504 bytes 2024-04-24T22:44:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 2018000896 bytes 2024-04-24T22:44:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 2018608128 bytes 2024-04-24T22:44:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 2069968896 bytes 2024-04-24T22:44:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 2137082880 bytes 2024-04-24T22:44:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2266852352 bytes 2024-04-24T22:44:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2324965376 bytes 2024-04-24T22:44:06.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2401857536 bytes 2024-04-24T22:44:06.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2452798464 bytes 2024-04-24T22:44:06.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2507839488 bytes 2024-04-24T22:44:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2508441600 bytes 2024-04-24T22:44:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2556322816 bytes 2024-04-24T22:44:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2682018816 bytes 2024-04-24T22:44:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2719636480 bytes 2024-04-24T22:44:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2754836480 bytes 2024-04-24T22:44:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2791972864 bytes 2024-04-24T22:44:07.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2823411712 bytes 2024-04-24T22:44:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2897957888 bytes 2024-04-24T22:44:07.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2927126528 bytes 2024-04-24T22:44:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2971270144 bytes 2024-04-24T22:44:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 3004502016 bytes 2024-04-24T22:44:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 3022525440 bytes 2024-04-24T22:44:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 3110820864 bytes 2024-04-24T22:44:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 3112500224 bytes 2024-04-24T22:44:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 3157374976 bytes 2024-04-24T22:44:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 3211035648 bytes 2024-04-24T22:44:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 3326362624 bytes 2024-04-24T22:44:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 3409715200 bytes 2024-04-24T22:44:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 3427565568 bytes 2024-04-24T22:44:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 3513011200 bytes 2024-04-24T22:44:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 3518579712 bytes 2024-04-24T22:44:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 3544069120 bytes 2024-04-24T22:44:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 3624554496 bytes 2024-04-24T22:44:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 3709195264 bytes 2024-04-24T22:44:10.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 3728375808 bytes 2024-04-24T22:44:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:44:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:44:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:44:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:44:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:44:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_30 2024-04-24T22:44:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2024-04-24T22:44:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T22:44:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:44:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:44:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_30 128M 2024-04-24T22:44:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T22:44:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:44:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:44:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:44:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_30 2024-04-24T22:44:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:44:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2024-04-24T22:44:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T22:44:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:44:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2024-04-24T22:44:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:44:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2024-04-24T22:44:11.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_30': 2024-04-24T22:44:11.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:44:11.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:44:11.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:44:11.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 13e3926981 2024-04-24T22:44:11.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.13e3926981 2024-04-24T22:44:11.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:44:11.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:44:11.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:44:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:44:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:44:10 2024 2024-04-24T22:44:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:44:10 2024 2024-04-24T22:44:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:44:10 2024 2024-04-24T22:44:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:44:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:44:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 4cd18b79-d4e1-402a-89e5-a62956f139ed 2024-04-24T22:44:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:44:11.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:44:11.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_30 2024-04-24T22:44:11.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:11.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:11.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T22:44:11.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:44:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=5 2024-04-24T22:44:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:44:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:44:11.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 2412544 bytes 2024-04-24T22:44:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 48748544 bytes 2024-04-24T22:44:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 108485632 bytes 2024-04-24T22:44:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 124740608 bytes 2024-04-24T22:44:11.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 150693888 bytes 2024-04-24T22:44:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 222184448 bytes 2024-04-24T22:44:12.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 254219264 bytes 2024-04-24T22:44:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 285305856 bytes 2024-04-24T22:44:12.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 376152064 bytes 2024-04-24T22:44:12.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 447591424 bytes 2024-04-24T22:44:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 518601728 bytes 2024-04-24T22:44:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 554184704 bytes 2024-04-24T22:44:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 579910656 bytes 2024-04-24T22:44:13.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 617734144 bytes 2024-04-24T22:44:13.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 671326208 bytes 2024-04-24T22:44:13.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 720964608 bytes 2024-04-24T22:44:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 782353408 bytes 2024-04-24T22:44:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 802673664 bytes 2024-04-24T22:44:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 817785856 bytes 2024-04-24T22:44:14.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 861491200 bytes 2024-04-24T22:44:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 944876544 bytes 2024-04-24T22:44:14.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 1028549632 bytes 2024-04-24T22:44:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1053555712 bytes 2024-04-24T22:44:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1094810624 bytes 2024-04-24T22:44:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1158588416 bytes 2024-04-24T22:44:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1179338752 bytes 2024-04-24T22:44:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1219996672 bytes 2024-04-24T22:44:15.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1239141376 bytes 2024-04-24T22:44:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1294521344 bytes 2024-04-24T22:44:15.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1368512512 bytes 2024-04-24T22:44:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1450596352 bytes 2024-04-24T22:44:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1451218944 bytes 2024-04-24T22:44:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1461485568 bytes 2024-04-24T22:44:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:44:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:44:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:44:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:44:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:44:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_31 2024-04-24T22:44:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2024-04-24T22:44:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T22:44:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:44:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:44:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_31 128M 2024-04-24T22:44:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T22:44:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:44:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:44:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:44:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_31 2024-04-24T22:44:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:44:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2024-04-24T22:44:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T22:44:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:44:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2024-04-24T22:44:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:44:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2024-04-24T22:44:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_31': 2024-04-24T22:44:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:44:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:44:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:44:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 13efcb5f746c 2024-04-24T22:44:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.13efcb5f746c 2024-04-24T22:44:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:44:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:44:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:44:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:44:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:44:16 2024 2024-04-24T22:44:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:44:16 2024 2024-04-24T22:44:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:44:16 2024 2024-04-24T22:44:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:44:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:44:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: e73f5c22-d59e-4e21-8ef5-f803b29356cb 2024-04-24T22:44:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:44:17.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:44:17.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_31 2024-04-24T22:44:17.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:17.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:17.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T22:44:17.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:44:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=12 2024-04-24T22:44:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:44:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:44:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 23351296 bytes 2024-04-24T22:44:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 46337024 bytes 2024-04-24T22:44:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 89066496 bytes 2024-04-24T22:44:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 154278912 bytes 2024-04-24T22:44:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 197349376 bytes 2024-04-24T22:44:18.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 286256128 bytes 2024-04-24T22:44:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 314299392 bytes 2024-04-24T22:44:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 399406080 bytes 2024-04-24T22:44:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 449980416 bytes 2024-04-24T22:44:18.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 496384000 bytes 2024-04-24T22:44:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 520156160 bytes 2024-04-24T22:44:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 584606720 bytes 2024-04-24T22:44:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 673884160 bytes 2024-04-24T22:44:19.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 706754560 bytes 2024-04-24T22:44:19.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 752968704 bytes 2024-04-24T22:44:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 774189056 bytes 2024-04-24T22:44:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 794544128 bytes 2024-04-24T22:44:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 810498048 bytes 2024-04-24T22:44:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 843913216 bytes 2024-04-24T22:44:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 862916608 bytes 2024-04-24T22:44:20.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 886917120 bytes 2024-04-24T22:44:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 899189760 bytes 2024-04-24T22:44:20.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 951163904 bytes 2024-04-24T22:44:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1035122688 bytes 2024-04-24T22:44:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1035721728 bytes 2024-04-24T22:44:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1077629952 bytes 2024-04-24T22:44:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1150256128 bytes 2024-04-24T22:44:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1183339520 bytes 2024-04-24T22:44:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1215431680 bytes 2024-04-24T22:44:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1315212288 bytes 2024-04-24T22:44:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1382090752 bytes 2024-04-24T22:44:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1400466432 bytes 2024-04-24T22:44:22.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1427179520 bytes 2024-04-24T22:44:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1484986368 bytes 2024-04-24T22:44:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1535905792 bytes 2024-04-24T22:44:22.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1599653888 bytes 2024-04-24T22:44:22.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1648360448 bytes 2024-04-24T22:44:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1664491520 bytes 2024-04-24T22:44:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1689857024 bytes 2024-04-24T22:44:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1718068224 bytes 2024-04-24T22:44:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1762781184 bytes 2024-04-24T22:44:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1779708928 bytes 2024-04-24T22:44:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1869248512 bytes 2024-04-24T22:44:23.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1895442432 bytes 2024-04-24T22:44:23.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1933331456 bytes 2024-04-24T22:44:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1957398528 bytes 2024-04-24T22:44:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1982413824 bytes 2024-04-24T22:44:24.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2025154560 bytes 2024-04-24T22:44:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2066989056 bytes 2024-04-24T22:44:24.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2104037376 bytes 2024-04-24T22:44:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2108483584 bytes 2024-04-24T22:44:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2168769536 bytes 2024-04-24T22:44:24.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2213787648 bytes 2024-04-24T22:44:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2298531840 bytes 2024-04-24T22:44:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2348105728 bytes 2024-04-24T22:44:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2390694912 bytes 2024-04-24T22:44:25.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2397469696 bytes 2024-04-24T22:44:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2464374784 bytes 2024-04-24T22:44:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2485255168 bytes 2024-04-24T22:44:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2506696704 bytes 2024-04-24T22:44:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2538110976 bytes 2024-04-24T22:44:26.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2543462400 bytes 2024-04-24T22:44:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2568412160 bytes 2024-04-24T22:44:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2659641344 bytes 2024-04-24T22:44:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2662733824 bytes 2024-04-24T22:44:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2696300544 bytes 2024-04-24T22:44:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2773300224 bytes 2024-04-24T22:44:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2819548160 bytes 2024-04-24T22:44:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2851937280 bytes 2024-04-24T22:44:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 2908492800 bytes 2024-04-24T22:44:27.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 2925577216 bytes 2024-04-24T22:44:27.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 3012614144 bytes 2024-04-24T22:44:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 3058266112 bytes 2024-04-24T22:44:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 3099380736 bytes 2024-04-24T22:44:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 3119935488 bytes 2024-04-24T22:44:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 3176316928 bytes 2024-04-24T22:44:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 3196289024 bytes 2024-04-24T22:44:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 3220167680 bytes 2024-04-24T22:44:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7900 IOs, 3235252224 bytes 2024-04-24T22:44:28.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8000 IOs, 3235875840 bytes 2024-04-24T22:44:28.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8100 IOs, 3261043712 bytes 2024-04-24T22:44:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8200 IOs, 3338104832 bytes 2024-04-24T22:44:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8300 IOs, 3395042304 bytes 2024-04-24T22:44:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:44:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:44:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:44:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:44:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:44:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_32 2024-04-24T22:44:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2024-04-24T22:44:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T22:44:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:44:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:44:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_32 128M 2024-04-24T22:44:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T22:44:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:44:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:44:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:44:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_32 2024-04-24T22:44:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:44:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2024-04-24T22:44:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T22:44:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:44:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2024-04-24T22:44:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:44:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2024-04-24T22:44:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_32': 2024-04-24T22:44:30.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:44:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:44:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:44:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 13fb98e3d412 2024-04-24T22:44:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.13fb98e3d412 2024-04-24T22:44:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:44:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:44:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:44:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:44:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:44:29 2024 2024-04-24T22:44:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:44:29 2024 2024-04-24T22:44:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:44:29 2024 2024-04-24T22:44:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:44:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:44:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: f185f517-ef74-4863-98b2-5700a59468b6 2024-04-24T22:44:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:44:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:44:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_32 2024-04-24T22:44:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T22:44:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:44:30.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=6 2024-04-24T22:44:30.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:44:30.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:44:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 81223680 bytes 2024-04-24T22:44:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 188674048 bytes 2024-04-24T22:44:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 235246592 bytes 2024-04-24T22:44:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 251718656 bytes 2024-04-24T22:44:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 315052032 bytes 2024-04-24T22:44:31.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 331569152 bytes 2024-04-24T22:44:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 426564608 bytes 2024-04-24T22:44:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 442360832 bytes 2024-04-24T22:44:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 456678400 bytes 2024-04-24T22:44:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 463793152 bytes 2024-04-24T22:44:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 493600768 bytes 2024-04-24T22:44:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 556004352 bytes 2024-04-24T22:44:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 620438528 bytes 2024-04-24T22:44:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 649785344 bytes 2024-04-24T22:44:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 671316992 bytes 2024-04-24T22:44:32.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 719190016 bytes 2024-04-24T22:44:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 749102080 bytes 2024-04-24T22:44:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 805146624 bytes 2024-04-24T22:44:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 848152576 bytes 2024-04-24T22:44:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 872198144 bytes 2024-04-24T22:44:33.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 895514624 bytes 2024-04-24T22:44:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 896119808 bytes 2024-04-24T22:44:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1000557568 bytes 2024-04-24T22:44:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1019604992 bytes 2024-04-24T22:44:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1076051968 bytes 2024-04-24T22:44:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1118702592 bytes 2024-04-24T22:44:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1131293696 bytes 2024-04-24T22:44:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1220360192 bytes 2024-04-24T22:44:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1262052352 bytes 2024-04-24T22:44:34.637 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-24T22:44:34.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-24T22:44:34.638 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-04-24T22:44:34.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T22:44:34.636+0000 7fe28f845700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 59468) UID: 1000 2024-04-24T22:44:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1294928896 bytes 2024-04-24T22:44:34.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:44:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1396427776 bytes 2024-04-24T22:44:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1442072576 bytes 2024-04-24T22:44:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1470008320 bytes 2024-04-24T22:44:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1520358400 bytes 2024-04-24T22:44:35.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1523867648 bytes 2024-04-24T22:44:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1543361536 bytes 2024-04-24T22:44:35.639 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-24T22:44:35.639 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:44:35.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:44:35.640 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-24T22:44:35.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-24T22:44:35.640 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-24T22:44:35.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-24T22:44:35.651 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2024-04-24T22:44:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1648105472 bytes 2024-04-24T22:44:35.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1699970048 bytes 2024-04-24T22:44:36.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1738382336 bytes 2024-04-24T22:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_33 2024-04-24T22:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2024-04-24T22:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T22:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:44:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:44:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_33 128M 2024-04-24T22:44:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T22:44:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:44:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:44:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:44:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_33 2024-04-24T22:44:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:44:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2024-04-24T22:44:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T22:44:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:44:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2024-04-24T22:44:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:44:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2024-04-24T22:44:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_33': 2024-04-24T22:44:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:44:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:44:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:44:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 140d88a16bd7 2024-04-24T22:44:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.140d88a16bd7 2024-04-24T22:44:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:44:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:44:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:44:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:44:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:44:36 2024 2024-04-24T22:44:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:44:36 2024 2024-04-24T22:44:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:44:36 2024 2024-04-24T22:44:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:44:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:44:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 9907f064-0835-4f4a-86eb-5688b009e4d7 2024-04-24T22:44:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:44:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:44:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_33 2024-04-24T22:44:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T22:44:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:44:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=11 2024-04-24T22:44:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:44:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:44:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 25758720 bytes 2024-04-24T22:44:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 34945024 bytes 2024-04-24T22:44:37.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 101018624 bytes 2024-04-24T22:44:38.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 260747264 bytes 2024-04-24T22:44:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 280386560 bytes 2024-04-24T22:44:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 344883200 bytes 2024-04-24T22:44:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 387511296 bytes 2024-04-24T22:44:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 420211712 bytes 2024-04-24T22:44:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 492226560 bytes 2024-04-24T22:44:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 494798848 bytes 2024-04-24T22:44:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 535170048 bytes 2024-04-24T22:44:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 565337088 bytes 2024-04-24T22:44:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 607341568 bytes 2024-04-24T22:44:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 645101568 bytes 2024-04-24T22:44:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 673601536 bytes 2024-04-24T22:44:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 724488192 bytes 2024-04-24T22:44:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 758082560 bytes 2024-04-24T22:44:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 823252992 bytes 2024-04-24T22:44:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 904242176 bytes 2024-04-24T22:44:40.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 924921856 bytes 2024-04-24T22:44:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 1033064448 bytes 2024-04-24T22:44:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 1056447488 bytes 2024-04-24T22:44:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1096552448 bytes 2024-04-24T22:44:41.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1132703744 bytes 2024-04-24T22:44:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1133296640 bytes 2024-04-24T22:44:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1154595840 bytes 2024-04-24T22:44:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1217480704 bytes 2024-04-24T22:44:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1296140288 bytes 2024-04-24T22:44:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1298545664 bytes 2024-04-24T22:44:41.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1300419584 bytes 2024-04-24T22:44:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1354975232 bytes 2024-04-24T22:44:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1370954752 bytes 2024-04-24T22:44:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1399336960 bytes 2024-04-24T22:44:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1439644672 bytes 2024-04-24T22:44:42.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1475539968 bytes 2024-04-24T22:44:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1499897856 bytes 2024-04-24T22:44:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1525537792 bytes 2024-04-24T22:44:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1529860096 bytes 2024-04-24T22:44:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1539437568 bytes 2024-04-24T22:44:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1611640832 bytes 2024-04-24T22:44:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1685888000 bytes 2024-04-24T22:44:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1757625344 bytes 2024-04-24T22:44:43.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1818697728 bytes 2024-04-24T22:44:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1834192896 bytes 2024-04-24T22:44:44.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1935759360 bytes 2024-04-24T22:44:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 2023068672 bytes 2024-04-24T22:44:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 2092318720 bytes 2024-04-24T22:44:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2116061184 bytes 2024-04-24T22:44:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2170787840 bytes 2024-04-24T22:44:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2261708800 bytes 2024-04-24T22:44:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2262315008 bytes 2024-04-24T22:44:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2305442816 bytes 2024-04-24T22:44:45.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2419251200 bytes 2024-04-24T22:44:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2431325184 bytes 2024-04-24T22:44:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2478209024 bytes 2024-04-24T22:44:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2529152000 bytes 2024-04-24T22:44:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2568941568 bytes 2024-04-24T22:44:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2589192192 bytes 2024-04-24T22:44:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2629233664 bytes 2024-04-24T22:44:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2656439296 bytes 2024-04-24T22:44:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2681014272 bytes 2024-04-24T22:44:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2744679424 bytes 2024-04-24T22:44:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2773329920 bytes 2024-04-24T22:44:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2795072512 bytes 2024-04-24T22:44:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2887931904 bytes 2024-04-24T22:44:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2897524736 bytes 2024-04-24T22:44:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2909990912 bytes 2024-04-24T22:44:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2939809792 bytes 2024-04-24T22:44:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2976457728 bytes 2024-04-24T22:44:47.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 3041332224 bytes 2024-04-24T22:44:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 3092641792 bytes 2024-04-24T22:44:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 3156623360 bytes 2024-04-24T22:44:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 3198060544 bytes 2024-04-24T22:44:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:44:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:44:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:44:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:44:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:44:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_34 2024-04-24T22:44:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2024-04-24T22:44:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T22:44:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:44:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:44:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_34 128M 2024-04-24T22:44:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T22:44:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:44:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:44:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:44:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_34 2024-04-24T22:44:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:44:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2024-04-24T22:44:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T22:44:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:44:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2024-04-24T22:44:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:44:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2024-04-24T22:44:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_34': 2024-04-24T22:44:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:44:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:44:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:44:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 1419e4db3bbc 2024-04-24T22:44:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.1419e4db3bbc 2024-04-24T22:44:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:44:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:44:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:44:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:44:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:44:48 2024 2024-04-24T22:44:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:44:48 2024 2024-04-24T22:44:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:44:48 2024 2024-04-24T22:44:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:44:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:44:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 8081cb7e-e5d5-4fb7-8838-62c6156ec6a4 2024-04-24T22:44:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:44:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:44:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_34 2024-04-24T22:44:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T22:44:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:44:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=5 2024-04-24T22:44:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:44:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:44:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 19382272 bytes 2024-04-24T22:44:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 86634496 bytes 2024-04-24T22:44:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 105119744 bytes 2024-04-24T22:44:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 225865728 bytes 2024-04-24T22:44:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 261771264 bytes 2024-04-24T22:44:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 272829440 bytes 2024-04-24T22:44:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 311993344 bytes 2024-04-24T22:44:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 332920832 bytes 2024-04-24T22:44:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 353694720 bytes 2024-04-24T22:44:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 403681280 bytes 2024-04-24T22:44:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 415940608 bytes 2024-04-24T22:44:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 459040768 bytes 2024-04-24T22:44:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 518980608 bytes 2024-04-24T22:44:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 549579776 bytes 2024-04-24T22:44:51.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 605375488 bytes 2024-04-24T22:44:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 607844352 bytes 2024-04-24T22:44:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 653536256 bytes 2024-04-24T22:44:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 681041920 bytes 2024-04-24T22:44:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 695925760 bytes 2024-04-24T22:44:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 738747392 bytes 2024-04-24T22:44:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 747441152 bytes 2024-04-24T22:44:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 794304512 bytes 2024-04-24T22:44:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 890654720 bytes 2024-04-24T22:44:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 910462976 bytes 2024-04-24T22:44:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 935872512 bytes 2024-04-24T22:44:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 988278784 bytes 2024-04-24T22:44:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1011992576 bytes 2024-04-24T22:44:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1056824320 bytes 2024-04-24T22:44:53.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1121574912 bytes 2024-04-24T22:44:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1169551360 bytes 2024-04-24T22:44:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1251861504 bytes 2024-04-24T22:44:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1263606784 bytes 2024-04-24T22:44:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1309747200 bytes 2024-04-24T22:44:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1449616384 bytes 2024-04-24T22:44:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:44:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:44:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:44:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:44:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:44:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_35 2024-04-24T22:44:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2024-04-24T22:44:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T22:44:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:44:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:44:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_35 128M 2024-04-24T22:44:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T22:44:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:44:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:44:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:44:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:44:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_35 2024-04-24T22:44:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:44:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2024-04-24T22:44:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T22:44:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:44:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2024-04-24T22:44:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:44:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2024-04-24T22:44:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_35': 2024-04-24T22:44:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:44:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:44:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:44:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 1425c75ebdda 2024-04-24T22:44:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.1425c75ebdda 2024-04-24T22:44:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:44:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:44:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:44:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:44:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:44:54 2024 2024-04-24T22:44:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:44:54 2024 2024-04-24T22:44:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:44:54 2024 2024-04-24T22:44:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:44:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:44:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 77b74900-6e9c-4b05-b994-e2ce65e571fa 2024-04-24T22:44:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:44:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:44:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_35 2024-04-24T22:44:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:44:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:44:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T22:44:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:44:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=11 2024-04-24T22:44:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:44:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:44:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 25713664 bytes 2024-04-24T22:44:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 50213888 bytes 2024-04-24T22:44:55.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 83177472 bytes 2024-04-24T22:44:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 128855040 bytes 2024-04-24T22:44:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 169879552 bytes 2024-04-24T22:44:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 211223552 bytes 2024-04-24T22:44:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 277749760 bytes 2024-04-24T22:44:56.412 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8943326855698092) > weight (0.25) 2024-04-24T22:44:56.412 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6427411536493015) > weight (0.25) 2024-04-24T22:44:56.412 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.30529795776642454) > weight (0.25) 2024-04-24T22:44:56.412 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-04-24T22:44:56.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-04-24T22:44:56.412 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-04-24T22:44:56.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T22:44:56.411+0000 7f0ff1051700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 52364) UID: 1000 2024-04-24T22:44:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 298889216 bytes 2024-04-24T22:44:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 304254976 bytes 2024-04-24T22:44:56.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:44:56.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 357986304 bytes 2024-04-24T22:44:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 396569600 bytes 2024-04-24T22:44:56.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 433128448 bytes 2024-04-24T22:44:57.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 458490880 bytes 2024-04-24T22:44:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 501252096 bytes 2024-04-24T22:44:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 527417344 bytes 2024-04-24T22:44:57.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 570700800 bytes 2024-04-24T22:44:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 666646528 bytes 2024-04-24T22:44:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 731558912 bytes 2024-04-24T22:44:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 759353344 bytes 2024-04-24T22:44:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 772147200 bytes 2024-04-24T22:44:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 825972736 bytes 2024-04-24T22:44:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 826568704 bytes 2024-04-24T22:44:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 899903488 bytes 2024-04-24T22:44:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 945876992 bytes 2024-04-24T22:44:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 989440000 bytes 2024-04-24T22:44:58.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1032682496 bytes 2024-04-24T22:44:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1038868480 bytes 2024-04-24T22:44:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1088036864 bytes 2024-04-24T22:44:59.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1128432640 bytes 2024-04-24T22:44:59.413 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-04-24T22:44:59.413 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:44:59.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:44:59.414 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-04-24T22:44:59.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-04-24T22:44:59.414 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-04-24T22:44:59.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-04-24T22:44:59.422 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2024-04-24T22:44:59.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1195796480 bytes 2024-04-24T22:44:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1219520512 bytes 2024-04-24T22:44:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1277568000 bytes 2024-04-24T22:44:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1338003456 bytes 2024-04-24T22:45:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1360006144 bytes 2024-04-24T22:45:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1377605632 bytes 2024-04-24T22:45:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1400108032 bytes 2024-04-24T22:45:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1463759872 bytes 2024-04-24T22:45:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1500237824 bytes 2024-04-24T22:45:00.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1544248320 bytes 2024-04-24T22:45:00.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1569189888 bytes 2024-04-24T22:45:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1595236352 bytes 2024-04-24T22:45:01.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1599280128 bytes 2024-04-24T22:45:01.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1624665088 bytes 2024-04-24T22:45:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1649916928 bytes 2024-04-24T22:45:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1736542208 bytes 2024-04-24T22:45:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1768811520 bytes 2024-04-24T22:45:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1813663744 bytes 2024-04-24T22:45:02.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1838649344 bytes 2024-04-24T22:45:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 1898573824 bytes 2024-04-24T22:45:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 1950807040 bytes 2024-04-24T22:45:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 1969017856 bytes 2024-04-24T22:45:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 1998047232 bytes 2024-04-24T22:45:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2057223168 bytes 2024-04-24T22:45:02.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2117124096 bytes 2024-04-24T22:45:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2180381696 bytes 2024-04-24T22:45:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2222856192 bytes 2024-04-24T22:45:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2255155200 bytes 2024-04-24T22:45:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2268069888 bytes 2024-04-24T22:45:03.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2282226688 bytes 2024-04-24T22:45:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2304025600 bytes 2024-04-24T22:45:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2310792192 bytes 2024-04-24T22:45:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2417724416 bytes 2024-04-24T22:45:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2474772480 bytes 2024-04-24T22:45:04.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2484014080 bytes 2024-04-24T22:45:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2557802496 bytes 2024-04-24T22:45:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2602553344 bytes 2024-04-24T22:45:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2656157696 bytes 2024-04-24T22:45:05.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2668810240 bytes 2024-04-24T22:45:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2723770368 bytes 2024-04-24T22:45:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 2861330432 bytes 2024-04-24T22:45:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 2881247232 bytes 2024-04-24T22:45:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 2960379904 bytes 2024-04-24T22:45:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 2994826240 bytes 2024-04-24T22:45:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 3024176128 bytes 2024-04-24T22:45:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 3055881216 bytes 2024-04-24T22:45:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:45:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:45:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:45:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:45:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:45:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_36 2024-04-24T22:45:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2024-04-24T22:45:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T22:45:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:45:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:45:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_36 128M 2024-04-24T22:45:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T22:45:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:45:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:45:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:45:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_36 2024-04-24T22:45:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:45:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2024-04-24T22:45:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T22:45:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:45:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2024-04-24T22:45:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:45:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2024-04-24T22:45:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_36': 2024-04-24T22:45:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:45:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:45:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:45:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 1439b69ba214 2024-04-24T22:45:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.1439b69ba214 2024-04-24T22:45:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:45:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:45:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:45:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:45:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:45:06 2024 2024-04-24T22:45:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:45:06 2024 2024-04-24T22:45:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:45:06 2024 2024-04-24T22:45:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:45:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:45:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: d616d270-b42d-464e-90d2-860a74167c56 2024-04-24T22:45:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:45:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:45:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_36 2024-04-24T22:45:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T22:45:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:45:07.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=7 2024-04-24T22:45:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:45:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:45:07.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 57256960 bytes 2024-04-24T22:45:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 102354944 bytes 2024-04-24T22:45:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 114544640 bytes 2024-04-24T22:45:08.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 120629248 bytes 2024-04-24T22:45:08.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 160620544 bytes 2024-04-24T22:45:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 222030848 bytes 2024-04-24T22:45:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 249921536 bytes 2024-04-24T22:45:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 327038976 bytes 2024-04-24T22:45:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 388269056 bytes 2024-04-24T22:45:08.926 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:45:08.923+0000 7f0f4ef7b700 -1 reset not still connected to 0x5613a93c6680 2024-04-24T22:45:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 416227328 bytes 2024-04-24T22:45:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 416800768 bytes 2024-04-24T22:45:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 417415168 bytes 2024-04-24T22:45:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 457892864 bytes 2024-04-24T22:45:09.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 503720960 bytes 2024-04-24T22:45:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 528067584 bytes 2024-04-24T22:45:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 584184832 bytes 2024-04-24T22:45:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 641676288 bytes 2024-04-24T22:45:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 749100032 bytes 2024-04-24T22:45:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 789128192 bytes 2024-04-24T22:45:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 853644288 bytes 2024-04-24T22:45:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 903261184 bytes 2024-04-24T22:45:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 973966336 bytes 2024-04-24T22:45:10.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1030088704 bytes 2024-04-24T22:45:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1128597504 bytes 2024-04-24T22:45:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1191999488 bytes 2024-04-24T22:45:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1230817280 bytes 2024-04-24T22:45:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1236487168 bytes 2024-04-24T22:45:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1298227200 bytes 2024-04-24T22:45:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1322550272 bytes 2024-04-24T22:45:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1347297280 bytes 2024-04-24T22:45:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1417083904 bytes 2024-04-24T22:45:12.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1479056384 bytes 2024-04-24T22:45:12.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1487645696 bytes 2024-04-24T22:45:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1574276096 bytes 2024-04-24T22:45:12.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1574909952 bytes 2024-04-24T22:45:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1575508992 bytes 2024-04-24T22:45:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1630085120 bytes 2024-04-24T22:45:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1636633600 bytes 2024-04-24T22:45:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1642769408 bytes 2024-04-24T22:45:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1674759168 bytes 2024-04-24T22:45:13.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1699876864 bytes 2024-04-24T22:45:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1769030656 bytes 2024-04-24T22:45:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1790577664 bytes 2024-04-24T22:45:13.759 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:45:13.751+0000 7f0f4ef7b700 -1 reset not still connected to 0x5613aa4f9d40 2024-04-24T22:45:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1806788608 bytes 2024-04-24T22:45:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1807523840 bytes 2024-04-24T22:45:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1888270336 bytes 2024-04-24T22:45:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1909917696 bytes 2024-04-24T22:45:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:45:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:45:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:45:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:45:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:45:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_37 2024-04-24T22:45:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2024-04-24T22:45:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T22:45:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:45:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:45:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_37 128M 2024-04-24T22:45:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T22:45:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:45:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:45:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:45:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_37 2024-04-24T22:45:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:45:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2024-04-24T22:45:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T22:45:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:45:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2024-04-24T22:45:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:45:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2024-04-24T22:45:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_37': 2024-04-24T22:45:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:45:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:45:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:45:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 14456dd40516 2024-04-24T22:45:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.14456dd40516 2024-04-24T22:45:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:45:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:45:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:45:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:45:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:45:14 2024 2024-04-24T22:45:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:45:14 2024 2024-04-24T22:45:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:45:14 2024 2024-04-24T22:45:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:45:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:45:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: a3bbbaa2-a609-401b-8231-3adcfdc73de8 2024-04-24T22:45:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:45:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:45:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_37 2024-04-24T22:45:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T22:45:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:45:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=9 2024-04-24T22:45:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:45:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:45:15.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 15083520 bytes 2024-04-24T22:45:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 39946240 bytes 2024-04-24T22:45:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 54462464 bytes 2024-04-24T22:45:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 74150912 bytes 2024-04-24T22:45:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 100249600 bytes 2024-04-24T22:45:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 118782976 bytes 2024-04-24T22:45:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 129640448 bytes 2024-04-24T22:45:16.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 152288256 bytes 2024-04-24T22:45:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 189586432 bytes 2024-04-24T22:45:16.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 275727360 bytes 2024-04-24T22:45:16.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 326827008 bytes 2024-04-24T22:45:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 418648064 bytes 2024-04-24T22:45:17.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 468215808 bytes 2024-04-24T22:45:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 504409088 bytes 2024-04-24T22:45:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 557133824 bytes 2024-04-24T22:45:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 564465664 bytes 2024-04-24T22:45:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 588040192 bytes 2024-04-24T22:45:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 644869120 bytes 2024-04-24T22:45:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 715058176 bytes 2024-04-24T22:45:18.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 732141568 bytes 2024-04-24T22:45:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 816438272 bytes 2024-04-24T22:45:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 851096576 bytes 2024-04-24T22:45:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 893601792 bytes 2024-04-24T22:45:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 902443008 bytes 2024-04-24T22:45:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 934583296 bytes 2024-04-24T22:45:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 955064320 bytes 2024-04-24T22:45:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 990595072 bytes 2024-04-24T22:45:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1041266688 bytes 2024-04-24T22:45:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1074724864 bytes 2024-04-24T22:45:19.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1151148032 bytes 2024-04-24T22:45:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1227958272 bytes 2024-04-24T22:45:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1311891456 bytes 2024-04-24T22:45:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1357140992 bytes 2024-04-24T22:45:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1382289408 bytes 2024-04-24T22:45:20.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1435611136 bytes 2024-04-24T22:45:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1495289856 bytes 2024-04-24T22:45:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1515283456 bytes 2024-04-24T22:45:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1557254144 bytes 2024-04-24T22:45:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1588728832 bytes 2024-04-24T22:45:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1645930496 bytes 2024-04-24T22:45:21.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1702349824 bytes 2024-04-24T22:45:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1788144640 bytes 2024-04-24T22:45:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1859137536 bytes 2024-04-24T22:45:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1918571520 bytes 2024-04-24T22:45:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1954597888 bytes 2024-04-24T22:45:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 2012351488 bytes 2024-04-24T22:45:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 2031984640 bytes 2024-04-24T22:45:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2088379392 bytes 2024-04-24T22:45:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2192320512 bytes 2024-04-24T22:45:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2234303488 bytes 2024-04-24T22:45:22.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2312734720 bytes 2024-04-24T22:45:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2377698304 bytes 2024-04-24T22:45:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2424713216 bytes 2024-04-24T22:45:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2483581952 bytes 2024-04-24T22:45:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2526538752 bytes 2024-04-24T22:45:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2527601664 bytes 2024-04-24T22:45:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2538516480 bytes 2024-04-24T22:45:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2598830080 bytes 2024-04-24T22:45:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2641410048 bytes 2024-04-24T22:45:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2659634176 bytes 2024-04-24T22:45:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2665031680 bytes 2024-04-24T22:45:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:45:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:45:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:45:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:45:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:45:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_38 2024-04-24T22:45:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2024-04-24T22:45:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T22:45:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:45:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:45:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_38 128M 2024-04-24T22:45:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T22:45:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:45:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:45:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:45:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_38 2024-04-24T22:45:24.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:45:24.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2024-04-24T22:45:24.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:24.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:24.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T22:45:24.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:45:24.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2024-04-24T22:45:25.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:45:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2024-04-24T22:45:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_38': 2024-04-24T22:45:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:45:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:45:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:45:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 1451a06c8cb 2024-04-24T22:45:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.1451a06c8cb 2024-04-24T22:45:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:45:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:45:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:45:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:45:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:45:24 2024 2024-04-24T22:45:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:45:24 2024 2024-04-24T22:45:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:45:24 2024 2024-04-24T22:45:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:45:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:45:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 0b0a4d2e-042e-4de3-97db-78d53010573b 2024-04-24T22:45:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:45:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:45:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_38 2024-04-24T22:45:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T22:45:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:45:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=5 2024-04-24T22:45:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:45:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:45:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 6410240 bytes 2024-04-24T22:45:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 47233024 bytes 2024-04-24T22:45:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 70213632 bytes 2024-04-24T22:45:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 153262080 bytes 2024-04-24T22:45:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 200216576 bytes 2024-04-24T22:45:26.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 278308864 bytes 2024-04-24T22:45:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 317272064 bytes 2024-04-24T22:45:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 318752768 bytes 2024-04-24T22:45:26.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 357953536 bytes 2024-04-24T22:45:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 388965376 bytes 2024-04-24T22:45:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 397788160 bytes 2024-04-24T22:45:26.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 416389120 bytes 2024-04-24T22:45:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 418006016 bytes 2024-04-24T22:45:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 433356800 bytes 2024-04-24T22:45:27.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 502309888 bytes 2024-04-24T22:45:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 537337856 bytes 2024-04-24T22:45:27.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 606481408 bytes 2024-04-24T22:45:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 656038912 bytes 2024-04-24T22:45:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 680152064 bytes 2024-04-24T22:45:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 757575680 bytes 2024-04-24T22:45:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 778523648 bytes 2024-04-24T22:45:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 817319936 bytes 2024-04-24T22:45:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 817947648 bytes 2024-04-24T22:45:28.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 884466688 bytes 2024-04-24T22:45:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 940616704 bytes 2024-04-24T22:45:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1031488512 bytes 2024-04-24T22:45:29.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1080531968 bytes 2024-04-24T22:45:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1121301504 bytes 2024-04-24T22:45:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1150816256 bytes 2024-04-24T22:45:29.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1221086208 bytes 2024-04-24T22:45:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1277530112 bytes 2024-04-24T22:45:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1349298176 bytes 2024-04-24T22:45:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1350654976 bytes 2024-04-24T22:45:30.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1366831104 bytes 2024-04-24T22:45:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:45:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:45:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:45:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:45:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:45:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_39 2024-04-24T22:45:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2024-04-24T22:45:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T22:45:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:45:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:45:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_39 128M 2024-04-24T22:45:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T22:45:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:45:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:45:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:45:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:30.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_39 2024-04-24T22:45:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:45:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2024-04-24T22:45:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T22:45:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:45:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2024-04-24T22:45:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:45:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2024-04-24T22:45:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_39': 2024-04-24T22:45:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:45:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:45:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:45:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 145d29973616 2024-04-24T22:45:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.145d29973616 2024-04-24T22:45:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:45:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:45:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:45:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:45:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:45:30 2024 2024-04-24T22:45:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:45:30 2024 2024-04-24T22:45:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:45:30 2024 2024-04-24T22:45:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:45:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:45:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 74beb076-4d18-4365-b1a8-d0b50a2536f5 2024-04-24T22:45:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:45:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:45:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_39 2024-04-24T22:45:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:31.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T22:45:31.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:45:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=6 2024-04-24T22:45:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:45:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:45:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 15399936 bytes 2024-04-24T22:45:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 64839680 bytes 2024-04-24T22:45:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 102714368 bytes 2024-04-24T22:45:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 134151168 bytes 2024-04-24T22:45:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 139392000 bytes 2024-04-24T22:45:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 188650496 bytes 2024-04-24T22:45:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 206657536 bytes 2024-04-24T22:45:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 257477632 bytes 2024-04-24T22:45:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 293175296 bytes 2024-04-24T22:45:32.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 338872320 bytes 2024-04-24T22:45:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 360085504 bytes 2024-04-24T22:45:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 375964672 bytes 2024-04-24T22:45:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 470966272 bytes 2024-04-24T22:45:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 505205760 bytes 2024-04-24T22:45:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 607457280 bytes 2024-04-24T22:45:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 646998016 bytes 2024-04-24T22:45:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 713189376 bytes 2024-04-24T22:45:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 746771456 bytes 2024-04-24T22:45:34.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 819505152 bytes 2024-04-24T22:45:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 844230656 bytes 2024-04-24T22:45:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 917140480 bytes 2024-04-24T22:45:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 929654784 bytes 2024-04-24T22:45:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 962741248 bytes 2024-04-24T22:45:34.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1031153664 bytes 2024-04-24T22:45:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1064085504 bytes 2024-04-24T22:45:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1096423424 bytes 2024-04-24T22:45:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1148818432 bytes 2024-04-24T22:45:35.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1204369408 bytes 2024-04-24T22:45:35.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1265948672 bytes 2024-04-24T22:45:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1314674688 bytes 2024-04-24T22:45:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1328136192 bytes 2024-04-24T22:45:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1396836352 bytes 2024-04-24T22:45:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1448916992 bytes 2024-04-24T22:45:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1525751808 bytes 2024-04-24T22:45:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1549225984 bytes 2024-04-24T22:45:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1628578816 bytes 2024-04-24T22:45:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1677074432 bytes 2024-04-24T22:45:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1750412288 bytes 2024-04-24T22:45:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1782097920 bytes 2024-04-24T22:45:37.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:45:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:45:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:45:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:45:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:45:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_40 2024-04-24T22:45:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2024-04-24T22:45:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T22:45:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:45:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:45:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_40 128M 2024-04-24T22:45:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T22:45:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:45:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:45:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:45:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_40 2024-04-24T22:45:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:45:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2024-04-24T22:45:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T22:45:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:45:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2024-04-24T22:45:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:45:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2024-04-24T22:45:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_40': 2024-04-24T22:45:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:45:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:45:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:45:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 14692687a623 2024-04-24T22:45:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.14692687a623 2024-04-24T22:45:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:45:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:45:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:45:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:45:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:45:37 2024 2024-04-24T22:45:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:45:37 2024 2024-04-24T22:45:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:45:37 2024 2024-04-24T22:45:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:45:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:45:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: b9d9d1d6-592b-471e-8736-169dcce0a795 2024-04-24T22:45:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:45:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:45:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_40 2024-04-24T22:45:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T22:45:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:45:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=13 2024-04-24T22:45:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:45:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:45:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 40691712 bytes 2024-04-24T22:45:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 123830272 bytes 2024-04-24T22:45:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 160273408 bytes 2024-04-24T22:45:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 209139712 bytes 2024-04-24T22:45:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 294383616 bytes 2024-04-24T22:45:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 327282688 bytes 2024-04-24T22:45:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 338276352 bytes 2024-04-24T22:45:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 388590592 bytes 2024-04-24T22:45:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 461142016 bytes 2024-04-24T22:45:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 491554816 bytes 2024-04-24T22:45:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 508296192 bytes 2024-04-24T22:45:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 548977664 bytes 2024-04-24T22:45:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 574945280 bytes 2024-04-24T22:45:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 598643712 bytes 2024-04-24T22:45:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 632867840 bytes 2024-04-24T22:45:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 686796800 bytes 2024-04-24T22:45:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 725478400 bytes 2024-04-24T22:45:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 875347968 bytes 2024-04-24T22:45:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 918735872 bytes 2024-04-24T22:45:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 970356736 bytes 2024-04-24T22:45:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 1045630976 bytes 2024-04-24T22:45:41.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 1103279104 bytes 2024-04-24T22:45:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1135763456 bytes 2024-04-24T22:45:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1164587008 bytes 2024-04-24T22:45:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1239314432 bytes 2024-04-24T22:45:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1256576000 bytes 2024-04-24T22:45:42.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1281201152 bytes 2024-04-24T22:45:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1286892544 bytes 2024-04-24T22:45:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1321783296 bytes 2024-04-24T22:45:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1340665856 bytes 2024-04-24T22:45:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1378875392 bytes 2024-04-24T22:45:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1459367936 bytes 2024-04-24T22:45:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1488053248 bytes 2024-04-24T22:45:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1530604544 bytes 2024-04-24T22:45:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1554328576 bytes 2024-04-24T22:45:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1611419648 bytes 2024-04-24T22:45:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1647807488 bytes 2024-04-24T22:45:43.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1704601600 bytes 2024-04-24T22:45:44.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1756304384 bytes 2024-04-24T22:45:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1785122816 bytes 2024-04-24T22:45:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1857494016 bytes 2024-04-24T22:45:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1951907840 bytes 2024-04-24T22:45:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1969373184 bytes 2024-04-24T22:45:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1997648896 bytes 2024-04-24T22:45:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 2003574784 bytes 2024-04-24T22:45:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 2047868928 bytes 2024-04-24T22:45:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 2121765888 bytes 2024-04-24T22:45:45.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2197305344 bytes 2024-04-24T22:45:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2262468608 bytes 2024-04-24T22:45:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2283282432 bytes 2024-04-24T22:45:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2314607616 bytes 2024-04-24T22:45:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2351790080 bytes 2024-04-24T22:45:46.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2370729984 bytes 2024-04-24T22:45:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2451369984 bytes 2024-04-24T22:45:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2490618880 bytes 2024-04-24T22:45:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2529588224 bytes 2024-04-24T22:45:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2539766784 bytes 2024-04-24T22:45:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2600172544 bytes 2024-04-24T22:45:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2600738816 bytes 2024-04-24T22:45:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2639149056 bytes 2024-04-24T22:45:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2686024704 bytes 2024-04-24T22:45:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2695888896 bytes 2024-04-24T22:45:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2696514560 bytes 2024-04-24T22:45:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2782485504 bytes 2024-04-24T22:45:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2800386048 bytes 2024-04-24T22:45:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2832401408 bytes 2024-04-24T22:45:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2848455680 bytes 2024-04-24T22:45:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2896560128 bytes 2024-04-24T22:45:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2912971776 bytes 2024-04-24T22:45:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 3009171456 bytes 2024-04-24T22:45:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 3121192960 bytes 2024-04-24T22:45:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 3145196544 bytes 2024-04-24T22:45:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 3198673920 bytes 2024-04-24T22:45:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 3216671744 bytes 2024-04-24T22:45:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 3251079168 bytes 2024-04-24T22:45:49.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 3288582144 bytes 2024-04-24T22:45:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 3339609088 bytes 2024-04-24T22:45:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 3406273536 bytes 2024-04-24T22:45:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7900 IOs, 3453032448 bytes 2024-04-24T22:45:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8000 IOs, 3515648000 bytes 2024-04-24T22:45:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8100 IOs, 3532189696 bytes 2024-04-24T22:45:50.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8200 IOs, 3566790656 bytes 2024-04-24T22:45:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8300 IOs, 3569140736 bytes 2024-04-24T22:45:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8400 IOs, 3601247232 bytes 2024-04-24T22:45:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8500 IOs, 3659360256 bytes 2024-04-24T22:45:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8600 IOs, 3686125568 bytes 2024-04-24T22:45:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8700 IOs, 3728826368 bytes 2024-04-24T22:45:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8800 IOs, 3783132160 bytes 2024-04-24T22:45:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8900 IOs, 3811045376 bytes 2024-04-24T22:45:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9000 IOs, 3847823360 bytes 2024-04-24T22:45:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:45:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:45:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:45:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:45:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:45:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_41 2024-04-24T22:45:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2024-04-24T22:45:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T22:45:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:45:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:45:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_41 128M 2024-04-24T22:45:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T22:45:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:45:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:45:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:45:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:45:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_41 2024-04-24T22:45:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:45:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2024-04-24T22:45:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T22:45:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:45:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2024-04-24T22:45:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:45:52.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2024-04-24T22:45:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_41': 2024-04-24T22:45:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:45:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:45:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:45:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 1475a14b86ab 2024-04-24T22:45:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.1475a14b86ab 2024-04-24T22:45:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:45:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:45:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:45:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:45:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:45:51 2024 2024-04-24T22:45:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:45:51 2024 2024-04-24T22:45:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:45:51 2024 2024-04-24T22:45:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:45:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:45:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: f62d1554-3ea2-4d48-bd05-d49435911c52 2024-04-24T22:45:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:45:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:45:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_41 2024-04-24T22:45:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:45:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:45:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T22:45:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:45:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=9 2024-04-24T22:45:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:45:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:45:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 13106176 bytes 2024-04-24T22:45:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 42641408 bytes 2024-04-24T22:45:52.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 79545344 bytes 2024-04-24T22:45:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 126007296 bytes 2024-04-24T22:45:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 189334528 bytes 2024-04-24T22:45:53.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 190730240 bytes 2024-04-24T22:45:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 257211392 bytes 2024-04-24T22:45:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 261586944 bytes 2024-04-24T22:45:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 283917312 bytes 2024-04-24T22:45:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 301753344 bytes 2024-04-24T22:45:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 348298240 bytes 2024-04-24T22:45:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 415865856 bytes 2024-04-24T22:45:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 447670272 bytes 2024-04-24T22:45:54.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 469803008 bytes 2024-04-24T22:45:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 499234816 bytes 2024-04-24T22:45:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 564216832 bytes 2024-04-24T22:45:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 573216768 bytes 2024-04-24T22:45:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 584150016 bytes 2024-04-24T22:45:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 647067648 bytes 2024-04-24T22:45:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 724934656 bytes 2024-04-24T22:45:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 757341184 bytes 2024-04-24T22:45:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 876945408 bytes 2024-04-24T22:45:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 931697664 bytes 2024-04-24T22:45:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 966766592 bytes 2024-04-24T22:45:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1008515072 bytes 2024-04-24T22:45:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1030047744 bytes 2024-04-24T22:45:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1040476160 bytes 2024-04-24T22:45:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1048005632 bytes 2024-04-24T22:45:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1074170880 bytes 2024-04-24T22:45:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1096056832 bytes 2024-04-24T22:45:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1121464320 bytes 2024-04-24T22:45:56.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1156282368 bytes 2024-04-24T22:45:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1203233792 bytes 2024-04-24T22:45:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1246893056 bytes 2024-04-24T22:45:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1308237824 bytes 2024-04-24T22:45:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1334242304 bytes 2024-04-24T22:45:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1383186432 bytes 2024-04-24T22:45:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1428963328 bytes 2024-04-24T22:45:57.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1445140480 bytes 2024-04-24T22:45:57.651 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3285774301671214) > weight (0.25) 2024-04-24T22:45:57.651 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3926372209317809) > weight (0.25) 2024-04-24T22:45:57.652 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.2762886525655458) > weight (0.25) 2024-04-24T22:45:57.652 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8366005134396185) > weight (0.25) 2024-04-24T22:45:57.652 INFO:tasks.rbd_mirror_thrash:waiting for 114 secs before thrashing 2024-04-24T22:45:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1448038400 bytes 2024-04-24T22:45:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1473036288 bytes 2024-04-24T22:45:58.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1584428032 bytes 2024-04-24T22:45:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1623720960 bytes 2024-04-24T22:45:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1705689088 bytes 2024-04-24T22:45:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1714265088 bytes 2024-04-24T22:45:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1817583616 bytes 2024-04-24T22:45:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1929072640 bytes 2024-04-24T22:45:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1960260608 bytes 2024-04-24T22:45:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 1981978624 bytes 2024-04-24T22:45:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2014388224 bytes 2024-04-24T22:45:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2066682880 bytes 2024-04-24T22:45:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2098107392 bytes 2024-04-24T22:45:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2206823424 bytes 2024-04-24T22:46:00.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2343547904 bytes 2024-04-24T22:46:00.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2384794624 bytes 2024-04-24T22:46:00.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2407910400 bytes 2024-04-24T22:46:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2409743360 bytes 2024-04-24T22:46:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2451798016 bytes 2024-04-24T22:46:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2483150848 bytes 2024-04-24T22:46:00.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2539669504 bytes 2024-04-24T22:46:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2546452480 bytes 2024-04-24T22:46:01.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2551066624 bytes 2024-04-24T22:46:01.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2562643968 bytes 2024-04-24T22:46:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:46:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:46:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:46:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:46:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:46:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_42 2024-04-24T22:46:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2024-04-24T22:46:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T22:46:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:46:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:46:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_42 128M 2024-04-24T22:46:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T22:46:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:46:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:46:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:46:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_42 2024-04-24T22:46:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:46:01.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2024-04-24T22:46:01.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:01.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:01.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T22:46:01.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:46:01.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2024-04-24T22:46:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:46:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2024-04-24T22:46:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_42': 2024-04-24T22:46:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:46:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:46:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:46:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 1481da1c0513 2024-04-24T22:46:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.1481da1c0513 2024-04-24T22:46:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:46:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:46:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:46:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:46:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:46:01 2024 2024-04-24T22:46:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:46:01 2024 2024-04-24T22:46:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:46:01 2024 2024-04-24T22:46:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:46:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:46:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: abffceee-6d69-43b8-8792-bb1ec5129a1f 2024-04-24T22:46:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:46:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:46:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_42 2024-04-24T22:46:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T22:46:02.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:46:02.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=10 2024-04-24T22:46:02.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:46:02.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:46:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 16669696 bytes 2024-04-24T22:46:02.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 114384896 bytes 2024-04-24T22:46:02.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 117668864 bytes 2024-04-24T22:46:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 119168000 bytes 2024-04-24T22:46:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 156345344 bytes 2024-04-24T22:46:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 156947456 bytes 2024-04-24T22:46:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 221168640 bytes 2024-04-24T22:46:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 250622976 bytes 2024-04-24T22:46:03.422 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.32670588099827613) > weight (0.25) 2024-04-24T22:46:03.423 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8054049703588751) > weight (0.25) 2024-04-24T22:46:03.423 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.266709561988275) > weight (0.25) 2024-04-24T22:46:03.423 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9171362576678121) > weight (0.25) 2024-04-24T22:46:03.423 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2024-04-24T22:46:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 270005248 bytes 2024-04-24T22:46:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 305988608 bytes 2024-04-24T22:46:03.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 405238784 bytes 2024-04-24T22:46:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 425425920 bytes 2024-04-24T22:46:04.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 455982080 bytes 2024-04-24T22:46:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 497987584 bytes 2024-04-24T22:46:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 527318016 bytes 2024-04-24T22:46:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 543695872 bytes 2024-04-24T22:46:04.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 562586624 bytes 2024-04-24T22:46:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 615671808 bytes 2024-04-24T22:46:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 626628608 bytes 2024-04-24T22:46:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 627240960 bytes 2024-04-24T22:46:05.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 716441600 bytes 2024-04-24T22:46:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 762430464 bytes 2024-04-24T22:46:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 779780096 bytes 2024-04-24T22:46:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 821980160 bytes 2024-04-24T22:46:05.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 868606976 bytes 2024-04-24T22:46:05.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 869242880 bytes 2024-04-24T22:46:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 885747712 bytes 2024-04-24T22:46:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 942096384 bytes 2024-04-24T22:46:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 963895296 bytes 2024-04-24T22:46:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 974174208 bytes 2024-04-24T22:46:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1001889792 bytes 2024-04-24T22:46:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1052205056 bytes 2024-04-24T22:46:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1072493568 bytes 2024-04-24T22:46:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1114752000 bytes 2024-04-24T22:46:07.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1166717952 bytes 2024-04-24T22:46:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1215173632 bytes 2024-04-24T22:46:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1215777792 bytes 2024-04-24T22:46:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1255928832 bytes 2024-04-24T22:46:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1312424960 bytes 2024-04-24T22:46:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1420376064 bytes 2024-04-24T22:46:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1449963520 bytes 2024-04-24T22:46:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1519038464 bytes 2024-04-24T22:46:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1519647744 bytes 2024-04-24T22:46:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1612241920 bytes 2024-04-24T22:46:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1705755648 bytes 2024-04-24T22:46:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1729070080 bytes 2024-04-24T22:46:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1751483392 bytes 2024-04-24T22:46:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1786136576 bytes 2024-04-24T22:46:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 1797405696 bytes 2024-04-24T22:46:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 1833904128 bytes 2024-04-24T22:46:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 1846430720 bytes 2024-04-24T22:46:09.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 1919980544 bytes 2024-04-24T22:46:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 1921830912 bytes 2024-04-24T22:46:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 1983705088 bytes 2024-04-24T22:46:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2003563520 bytes 2024-04-24T22:46:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2121657344 bytes 2024-04-24T22:46:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2126471168 bytes 2024-04-24T22:46:10.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2182388736 bytes 2024-04-24T22:46:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2221528064 bytes 2024-04-24T22:46:11.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2244970496 bytes 2024-04-24T22:46:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2300871680 bytes 2024-04-24T22:46:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2337283072 bytes 2024-04-24T22:46:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2353550336 bytes 2024-04-24T22:46:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2373915648 bytes 2024-04-24T22:46:11.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2416891904 bytes 2024-04-24T22:46:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2462633984 bytes 2024-04-24T22:46:11.926 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T22:46:11.924+0000 7f05853c0700 -1 reset not still connected to 0x5619adb76c30 2024-04-24T22:46:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2519269376 bytes 2024-04-24T22:46:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2591978496 bytes 2024-04-24T22:46:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:46:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:46:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:46:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:46:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:46:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_43 2024-04-24T22:46:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2024-04-24T22:46:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T22:46:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:46:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:46:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_43 128M 2024-04-24T22:46:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T22:46:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:46:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:46:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:46:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_43 2024-04-24T22:46:12.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:46:12.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2024-04-24T22:46:12.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:12.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:12.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T22:46:12.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:46:12.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2024-04-24T22:46:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:46:13.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2024-04-24T22:46:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_43': 2024-04-24T22:46:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:46:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:46:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:46:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 148dce27dea6 2024-04-24T22:46:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.148dce27dea6 2024-04-24T22:46:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:46:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:46:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:46:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:46:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:46:12 2024 2024-04-24T22:46:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:46:12 2024 2024-04-24T22:46:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:46:12 2024 2024-04-24T22:46:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:46:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:46:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 50cbbefd-55ee-46b7-9b9e-c53db4b8ed79 2024-04-24T22:46:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:46:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:46:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_43 2024-04-24T22:46:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T22:46:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:46:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=14 2024-04-24T22:46:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:46:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:46:13.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 31503360 bytes 2024-04-24T22:46:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 32121856 bytes 2024-04-24T22:46:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 86818816 bytes 2024-04-24T22:46:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 186598400 bytes 2024-04-24T22:46:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 187208704 bytes 2024-04-24T22:46:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 238596096 bytes 2024-04-24T22:46:14.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 317894656 bytes 2024-04-24T22:46:14.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 398639104 bytes 2024-04-24T22:46:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 456763392 bytes 2024-04-24T22:46:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 494885888 bytes 2024-04-24T22:46:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 500005888 bytes 2024-04-24T22:46:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 538141696 bytes 2024-04-24T22:46:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 588803072 bytes 2024-04-24T22:46:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 638554112 bytes 2024-04-24T22:46:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 687899648 bytes 2024-04-24T22:46:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 737722368 bytes 2024-04-24T22:46:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 750032896 bytes 2024-04-24T22:46:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 774090752 bytes 2024-04-24T22:46:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 820355072 bytes 2024-04-24T22:46:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 835383296 bytes 2024-04-24T22:46:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 835997696 bytes 2024-04-24T22:46:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 881737728 bytes 2024-04-24T22:46:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 903603200 bytes 2024-04-24T22:46:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 928287744 bytes 2024-04-24T22:46:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 953641984 bytes 2024-04-24T22:46:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1019610112 bytes 2024-04-24T22:46:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1088256000 bytes 2024-04-24T22:46:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1118343168 bytes 2024-04-24T22:46:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1170761728 bytes 2024-04-24T22:46:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1210315776 bytes 2024-04-24T22:46:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1228453888 bytes 2024-04-24T22:46:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1255712768 bytes 2024-04-24T22:46:17.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1295880192 bytes 2024-04-24T22:46:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1348506624 bytes 2024-04-24T22:46:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1403417600 bytes 2024-04-24T22:46:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1439365120 bytes 2024-04-24T22:46:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1442590720 bytes 2024-04-24T22:46:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1471282176 bytes 2024-04-24T22:46:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1513308160 bytes 2024-04-24T22:46:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1582041088 bytes 2024-04-24T22:46:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1598531584 bytes 2024-04-24T22:46:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1601965056 bytes 2024-04-24T22:46:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1635723264 bytes 2024-04-24T22:46:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1680021504 bytes 2024-04-24T22:46:19.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1706661888 bytes 2024-04-24T22:46:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1744735232 bytes 2024-04-24T22:46:19.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1749444608 bytes 2024-04-24T22:46:20.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1795683328 bytes 2024-04-24T22:46:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 1814544384 bytes 2024-04-24T22:46:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 1860013056 bytes 2024-04-24T22:46:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 1895192576 bytes 2024-04-24T22:46:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 1944798208 bytes 2024-04-24T22:46:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 1945437184 bytes 2024-04-24T22:46:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 1965255680 bytes 2024-04-24T22:46:20.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2046219264 bytes 2024-04-24T22:46:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2046835712 bytes 2024-04-24T22:46:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2057394176 bytes 2024-04-24T22:46:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2081150976 bytes 2024-04-24T22:46:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2148546560 bytes 2024-04-24T22:46:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2210689024 bytes 2024-04-24T22:46:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2291888128 bytes 2024-04-24T22:46:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2348689408 bytes 2024-04-24T22:46:22.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2451412992 bytes 2024-04-24T22:46:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2487436288 bytes 2024-04-24T22:46:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2520107008 bytes 2024-04-24T22:46:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2544551936 bytes 2024-04-24T22:46:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2595019776 bytes 2024-04-24T22:46:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2636697600 bytes 2024-04-24T22:46:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2637321216 bytes 2024-04-24T22:46:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 2713403392 bytes 2024-04-24T22:46:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 2733747200 bytes 2024-04-24T22:46:23.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 2734367744 bytes 2024-04-24T22:46:23.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 2749782016 bytes 2024-04-24T22:46:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 2803080192 bytes 2024-04-24T22:46:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 2842166272 bytes 2024-04-24T22:46:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 2888954880 bytes 2024-04-24T22:46:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 2944150528 bytes 2024-04-24T22:46:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 3013348352 bytes 2024-04-24T22:46:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7900 IOs, 3066425344 bytes 2024-04-24T22:46:24.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8000 IOs, 3142773760 bytes 2024-04-24T22:46:24.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8100 IOs, 3203481600 bytes 2024-04-24T22:46:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8200 IOs, 3268619264 bytes 2024-04-24T22:46:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8300 IOs, 3291032576 bytes 2024-04-24T22:46:25.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8400 IOs, 3349072896 bytes 2024-04-24T22:46:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8500 IOs, 3424025600 bytes 2024-04-24T22:46:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8600 IOs, 3444366336 bytes 2024-04-24T22:46:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8700 IOs, 3446760448 bytes 2024-04-24T22:46:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8800 IOs, 3481827328 bytes 2024-04-24T22:46:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8900 IOs, 3501147136 bytes 2024-04-24T22:46:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9000 IOs, 3534397440 bytes 2024-04-24T22:46:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9100 IOs, 3562041344 bytes 2024-04-24T22:46:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9200 IOs, 3616095232 bytes 2024-04-24T22:46:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9300 IOs, 3639562240 bytes 2024-04-24T22:46:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9400 IOs, 3701781504 bytes 2024-04-24T22:46:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9500 IOs, 3734225920 bytes 2024-04-24T22:46:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9600 IOs, 3800350720 bytes 2024-04-24T22:46:27.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9700 IOs, 3886736384 bytes 2024-04-24T22:46:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:46:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:46:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:46:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:46:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:46:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_44 2024-04-24T22:46:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2024-04-24T22:46:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T22:46:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:46:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:46:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_44 128M 2024-04-24T22:46:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T22:46:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:46:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:46:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:46:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_44 2024-04-24T22:46:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:46:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2024-04-24T22:46:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T22:46:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:46:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2024-04-24T22:46:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:46:27.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2024-04-24T22:46:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_44': 2024-04-24T22:46:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:46:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:46:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:46:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 1499c312f096 2024-04-24T22:46:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.1499c312f096 2024-04-24T22:46:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:46:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:46:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:46:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:46:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:46:27 2024 2024-04-24T22:46:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:46:27 2024 2024-04-24T22:46:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:46:27 2024 2024-04-24T22:46:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:46:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:46:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 1cecaa75-0bd6-483a-abc1-814cd6c8b99b 2024-04-24T22:46:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:46:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:46:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_44 2024-04-24T22:46:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T22:46:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:46:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=7 2024-04-24T22:46:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:46:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:46:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 28546048 bytes 2024-04-24T22:46:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 105627648 bytes 2024-04-24T22:46:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 109855744 bytes 2024-04-24T22:46:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 168739840 bytes 2024-04-24T22:46:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 238972928 bytes 2024-04-24T22:46:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 285802496 bytes 2024-04-24T22:46:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 372512768 bytes 2024-04-24T22:46:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 387394560 bytes 2024-04-24T22:46:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 411224064 bytes 2024-04-24T22:46:29.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 442653696 bytes 2024-04-24T22:46:29.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 500984832 bytes 2024-04-24T22:46:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 539382784 bytes 2024-04-24T22:46:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 614629376 bytes 2024-04-24T22:46:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 679544832 bytes 2024-04-24T22:46:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 765855744 bytes 2024-04-24T22:46:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 822467584 bytes 2024-04-24T22:46:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 872821760 bytes 2024-04-24T22:46:30.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 902585344 bytes 2024-04-24T22:46:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 925982720 bytes 2024-04-24T22:46:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 1008344064 bytes 2024-04-24T22:46:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 1039005696 bytes 2024-04-24T22:46:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 1110631424 bytes 2024-04-24T22:46:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1183515648 bytes 2024-04-24T22:46:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1249858560 bytes 2024-04-24T22:46:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1302545408 bytes 2024-04-24T22:46:31.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1408923648 bytes 2024-04-24T22:46:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1447489536 bytes 2024-04-24T22:46:32.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1506955264 bytes 2024-04-24T22:46:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1535380480 bytes 2024-04-24T22:46:32.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1558912000 bytes 2024-04-24T22:46:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1572942848 bytes 2024-04-24T22:46:32.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1597675520 bytes 2024-04-24T22:46:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1605598208 bytes 2024-04-24T22:46:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1691815936 bytes 2024-04-24T22:46:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1816323072 bytes 2024-04-24T22:46:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1904712704 bytes 2024-04-24T22:46:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1942969344 bytes 2024-04-24T22:46:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1989810176 bytes 2024-04-24T22:46:33.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 2020417536 bytes 2024-04-24T22:46:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 2038464512 bytes 2024-04-24T22:46:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 2092026880 bytes 2024-04-24T22:46:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 2108786688 bytes 2024-04-24T22:46:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 2143563776 bytes 2024-04-24T22:46:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 2187253760 bytes 2024-04-24T22:46:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:46:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:46:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:46:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:46:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:46:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_45 2024-04-24T22:46:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2024-04-24T22:46:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T22:46:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:46:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:46:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_45 128M 2024-04-24T22:46:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T22:46:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:46:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:46:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:46:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_45 2024-04-24T22:46:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:46:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2024-04-24T22:46:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T22:46:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:46:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2024-04-24T22:46:35.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:46:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2024-04-24T22:46:35.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_45': 2024-04-24T22:46:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:46:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:46:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:46:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 14a565902cbc 2024-04-24T22:46:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.14a565902cbc 2024-04-24T22:46:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:46:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:46:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:46:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:46:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:46:34 2024 2024-04-24T22:46:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:46:34 2024 2024-04-24T22:46:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:46:34 2024 2024-04-24T22:46:35.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:46:35.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:46:35.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 94475b78-a054-4f7b-8225-9f06e6be39b9 2024-04-24T22:46:35.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:46:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:46:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_45 2024-04-24T22:46:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T22:46:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:46:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=10 2024-04-24T22:46:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:46:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:46:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 33433600 bytes 2024-04-24T22:46:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 34031616 bytes 2024-04-24T22:46:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 96949248 bytes 2024-04-24T22:46:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 120431616 bytes 2024-04-24T22:46:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 199887872 bytes 2024-04-24T22:46:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 219988992 bytes 2024-04-24T22:46:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 324400128 bytes 2024-04-24T22:46:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 378200064 bytes 2024-04-24T22:46:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 400293888 bytes 2024-04-24T22:46:36.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 447523840 bytes 2024-04-24T22:46:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 497427456 bytes 2024-04-24T22:46:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 536632320 bytes 2024-04-24T22:46:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 603461632 bytes 2024-04-24T22:46:37.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 635514880 bytes 2024-04-24T22:46:37.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 673477632 bytes 2024-04-24T22:46:37.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 722447360 bytes 2024-04-24T22:46:37.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 795699200 bytes 2024-04-24T22:46:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 817453056 bytes 2024-04-24T22:46:38.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 963038208 bytes 2024-04-24T22:46:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 963636224 bytes 2024-04-24T22:46:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 1008529408 bytes 2024-04-24T22:46:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 1009115136 bytes 2024-04-24T22:46:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1018409984 bytes 2024-04-24T22:46:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1089996800 bytes 2024-04-24T22:46:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1090613248 bytes 2024-04-24T22:46:39.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1147762688 bytes 2024-04-24T22:46:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1172670464 bytes 2024-04-24T22:46:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1230963712 bytes 2024-04-24T22:46:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1293260800 bytes 2024-04-24T22:46:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1331100672 bytes 2024-04-24T22:46:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1413911552 bytes 2024-04-24T22:46:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1446515712 bytes 2024-04-24T22:46:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1519880192 bytes 2024-04-24T22:46:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1553448960 bytes 2024-04-24T22:46:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1570797568 bytes 2024-04-24T22:46:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1571394560 bytes 2024-04-24T22:46:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1606556672 bytes 2024-04-24T22:46:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1607158784 bytes 2024-04-24T22:46:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1681795072 bytes 2024-04-24T22:46:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1796754432 bytes 2024-04-24T22:46:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1828348928 bytes 2024-04-24T22:46:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1842833408 bytes 2024-04-24T22:46:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1882979328 bytes 2024-04-24T22:46:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1897916416 bytes 2024-04-24T22:46:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1961109504 bytes 2024-04-24T22:46:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1996412928 bytes 2024-04-24T22:46:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1997024256 bytes 2024-04-24T22:46:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2052206592 bytes 2024-04-24T22:46:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2087140352 bytes 2024-04-24T22:46:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2116087808 bytes 2024-04-24T22:46:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2116722688 bytes 2024-04-24T22:46:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2151823360 bytes 2024-04-24T22:46:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2161883136 bytes 2024-04-24T22:46:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2226691072 bytes 2024-04-24T22:46:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2258893824 bytes 2024-04-24T22:46:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2284653568 bytes 2024-04-24T22:46:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2308361216 bytes 2024-04-24T22:46:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2331797504 bytes 2024-04-24T22:46:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2346654720 bytes 2024-04-24T22:46:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2396294144 bytes 2024-04-24T22:46:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2444091392 bytes 2024-04-24T22:46:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2478479360 bytes 2024-04-24T22:46:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2479080448 bytes 2024-04-24T22:46:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2509313024 bytes 2024-04-24T22:46:44.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2534219776 bytes 2024-04-24T22:46:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2599990272 bytes 2024-04-24T22:46:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2627346432 bytes 2024-04-24T22:46:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2665166848 bytes 2024-04-24T22:46:45.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2669317120 bytes 2024-04-24T22:46:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:46:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:46:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:46:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:46:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:46:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_46 2024-04-24T22:46:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2024-04-24T22:46:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T22:46:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:46:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:46:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_46 128M 2024-04-24T22:46:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T22:46:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:46:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:46:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:46:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_46 2024-04-24T22:46:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:46:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2024-04-24T22:46:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T22:46:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:46:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2024-04-24T22:46:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:46:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2024-04-24T22:46:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_46': 2024-04-24T22:46:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:46:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:46:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:46:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 14b1da8131d9 2024-04-24T22:46:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.14b1da8131d9 2024-04-24T22:46:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:46:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:46:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:46:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:46:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:46:45 2024 2024-04-24T22:46:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:46:45 2024 2024-04-24T22:46:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:46:45 2024 2024-04-24T22:46:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:46:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:46:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: aaca0d68-80e6-4367-94cf-716faf3d8e67 2024-04-24T22:46:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:46:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:46:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_46 2024-04-24T22:46:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T22:46:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:46:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=5 2024-04-24T22:46:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:46:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:46:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 62039040 bytes 2024-04-24T22:46:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 109707264 bytes 2024-04-24T22:46:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 169485312 bytes 2024-04-24T22:46:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 212794368 bytes 2024-04-24T22:46:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 271989760 bytes 2024-04-24T22:46:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 301619200 bytes 2024-04-24T22:46:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 351521792 bytes 2024-04-24T22:46:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 378078208 bytes 2024-04-24T22:46:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 432487424 bytes 2024-04-24T22:46:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 443380736 bytes 2024-04-24T22:46:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 493424640 bytes 2024-04-24T22:46:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 509676544 bytes 2024-04-24T22:46:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 537040896 bytes 2024-04-24T22:46:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 583689216 bytes 2024-04-24T22:46:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 613953536 bytes 2024-04-24T22:46:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 623933440 bytes 2024-04-24T22:46:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 624544768 bytes 2024-04-24T22:46:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 644536320 bytes 2024-04-24T22:46:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 702213120 bytes 2024-04-24T22:46:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 738510848 bytes 2024-04-24T22:46:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 771243008 bytes 2024-04-24T22:46:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 776062976 bytes 2024-04-24T22:46:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 776881152 bytes 2024-04-24T22:46:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 825455616 bytes 2024-04-24T22:46:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 924503040 bytes 2024-04-24T22:46:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1000458240 bytes 2024-04-24T22:46:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1021279232 bytes 2024-04-24T22:46:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1040678912 bytes 2024-04-24T22:46:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1111393280 bytes 2024-04-24T22:46:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1207023616 bytes 2024-04-24T22:46:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1303848960 bytes 2024-04-24T22:46:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1347319808 bytes 2024-04-24T22:46:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1356657664 bytes 2024-04-24T22:46:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1411459072 bytes 2024-04-24T22:46:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:46:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:46:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:46:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:46:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:46:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_47 2024-04-24T22:46:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2024-04-24T22:46:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T22:46:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:46:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:46:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_47 128M 2024-04-24T22:46:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T22:46:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:46:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:46:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:46:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_47 2024-04-24T22:46:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:46:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2024-04-24T22:46:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T22:46:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:46:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2024-04-24T22:46:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:46:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2024-04-24T22:46:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_47': 2024-04-24T22:46:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:46:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:46:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:46:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 14bd4c73f192 2024-04-24T22:46:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.14bd4c73f192 2024-04-24T22:46:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:46:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:46:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:46:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:46:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:46:51 2024 2024-04-24T22:46:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:46:51 2024 2024-04-24T22:46:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:46:51 2024 2024-04-24T22:46:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:46:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:46:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: b943dd73-ded7-434d-8abb-ad7480804db1 2024-04-24T22:46:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:46:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:46:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_47 2024-04-24T22:46:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T22:46:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:46:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=7 2024-04-24T22:46:52.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:46:52.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:46:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 70307840 bytes 2024-04-24T22:46:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 116149248 bytes 2024-04-24T22:46:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 174270464 bytes 2024-04-24T22:46:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 217729024 bytes 2024-04-24T22:46:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 282320896 bytes 2024-04-24T22:46:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 286098432 bytes 2024-04-24T22:46:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 308403200 bytes 2024-04-24T22:46:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 349677568 bytes 2024-04-24T22:46:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 385646592 bytes 2024-04-24T22:46:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 415332352 bytes 2024-04-24T22:46:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 448112640 bytes 2024-04-24T22:46:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 500803584 bytes 2024-04-24T22:46:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 544840704 bytes 2024-04-24T22:46:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 575356928 bytes 2024-04-24T22:46:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 626949120 bytes 2024-04-24T22:46:54.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 674266112 bytes 2024-04-24T22:46:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 715968512 bytes 2024-04-24T22:46:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 759346176 bytes 2024-04-24T22:46:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 770590720 bytes 2024-04-24T22:46:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 809977856 bytes 2024-04-24T22:46:55.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 849059840 bytes 2024-04-24T22:46:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 876801024 bytes 2024-04-24T22:46:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 922269696 bytes 2024-04-24T22:46:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 974892032 bytes 2024-04-24T22:46:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 983260160 bytes 2024-04-24T22:46:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1025282048 bytes 2024-04-24T22:46:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1055886336 bytes 2024-04-24T22:46:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1135514624 bytes 2024-04-24T22:46:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1142044672 bytes 2024-04-24T22:46:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1220569088 bytes 2024-04-24T22:46:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1240816640 bytes 2024-04-24T22:46:56.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1278053376 bytes 2024-04-24T22:46:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1295178752 bytes 2024-04-24T22:46:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1365700608 bytes 2024-04-24T22:46:57.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1366291456 bytes 2024-04-24T22:46:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1457779712 bytes 2024-04-24T22:46:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1555670016 bytes 2024-04-24T22:46:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1571038208 bytes 2024-04-24T22:46:57.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1605784576 bytes 2024-04-24T22:46:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1642516480 bytes 2024-04-24T22:46:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1697947648 bytes 2024-04-24T22:46:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1714382848 bytes 2024-04-24T22:46:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1751051264 bytes 2024-04-24T22:46:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1757621248 bytes 2024-04-24T22:46:58.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1779778560 bytes 2024-04-24T22:46:58.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1825815552 bytes 2024-04-24T22:46:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1866131456 bytes 2024-04-24T22:46:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1916542976 bytes 2024-04-24T22:46:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:46:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:46:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:46:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:46:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:46:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_48 2024-04-24T22:46:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2024-04-24T22:46:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T22:46:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:46:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:46:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_48 128M 2024-04-24T22:46:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T22:46:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:46:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:46:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:46:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:46:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_48 2024-04-24T22:46:59.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:46:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2024-04-24T22:46:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:46:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:46:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T22:46:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:46:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2024-04-24T22:47:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:47:00.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2024-04-24T22:47:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_48': 2024-04-24T22:47:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:47:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:47:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:47:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 14c9f32d83a9 2024-04-24T22:47:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.14c9f32d83a9 2024-04-24T22:47:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:47:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:47:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:47:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:47:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:46:59 2024 2024-04-24T22:47:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:46:59 2024 2024-04-24T22:47:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:46:59 2024 2024-04-24T22:47:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:47:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:47:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 4b92c784-d7e2-461f-9fc8-08abc3ca10c8 2024-04-24T22:47:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:47:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:47:00.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_48 2024-04-24T22:47:00.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:47:00.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:47:00.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T22:47:00.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:47:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=7 2024-04-24T22:47:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:47:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:47:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 25133056 bytes 2024-04-24T22:47:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 42437632 bytes 2024-04-24T22:47:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 112088064 bytes 2024-04-24T22:47:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 175935488 bytes 2024-04-24T22:47:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 214319104 bytes 2024-04-24T22:47:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 276865024 bytes 2024-04-24T22:47:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 293869568 bytes 2024-04-24T22:47:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 295672832 bytes 2024-04-24T22:47:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 337214464 bytes 2024-04-24T22:47:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 422721536 bytes 2024-04-24T22:47:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 456296448 bytes 2024-04-24T22:47:02.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 540920832 bytes 2024-04-24T22:47:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 578680832 bytes 2024-04-24T22:47:02.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 650287104 bytes 2024-04-24T22:47:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 656589824 bytes 2024-04-24T22:47:02.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 688418816 bytes 2024-04-24T22:47:02.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 701244416 bytes 2024-04-24T22:47:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 713992192 bytes 2024-04-24T22:47:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 785654784 bytes 2024-04-24T22:47:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 786260992 bytes 2024-04-24T22:47:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 820588544 bytes 2024-04-24T22:47:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 860486656 bytes 2024-04-24T22:47:03.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 898481152 bytes 2024-04-24T22:47:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 918091776 bytes 2024-04-24T22:47:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 918717440 bytes 2024-04-24T22:47:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 945951744 bytes 2024-04-24T22:47:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 988612608 bytes 2024-04-24T22:47:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1041395712 bytes 2024-04-24T22:47:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1091068928 bytes 2024-04-24T22:47:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1134854144 bytes 2024-04-24T22:47:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1199488000 bytes 2024-04-24T22:47:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1222543360 bytes 2024-04-24T22:47:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1223153664 bytes 2024-04-24T22:47:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1263537152 bytes 2024-04-24T22:47:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1265875968 bytes 2024-04-24T22:47:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1320400896 bytes 2024-04-24T22:47:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1382712320 bytes 2024-04-24T22:47:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1442298880 bytes 2024-04-24T22:47:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1494766592 bytes 2024-04-24T22:47:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1533633536 bytes 2024-04-24T22:47:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1578652672 bytes 2024-04-24T22:47:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1622917120 bytes 2024-04-24T22:47:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1720092672 bytes 2024-04-24T22:47:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1740395520 bytes 2024-04-24T22:47:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1778367488 bytes 2024-04-24T22:47:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1799815168 bytes 2024-04-24T22:47:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1830850560 bytes 2024-04-24T22:47:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1893324800 bytes 2024-04-24T22:47:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:47:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:47:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:47:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:47:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:47:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_49 2024-04-24T22:47:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2024-04-24T22:47:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:47:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:47:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:47:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:47:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T22:47:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:47:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:47:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:47:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:47:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_49 128M 2024-04-24T22:47:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:47:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:47:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:47:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:47:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T22:47:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:47:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:47:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:47:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:47:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:47:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_49 2024-04-24T22:47:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:47:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2024-04-24T22:47:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:47:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:47:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T22:47:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:47:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2024-04-24T22:47:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:47:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2024-04-24T22:47:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_49': 2024-04-24T22:47:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:47:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:47:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:47:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 14d5783d69a7 2024-04-24T22:47:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.14d5783d69a7 2024-04-24T22:47:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:47:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:47:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:47:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:47:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:47:07 2024 2024-04-24T22:47:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:47:07 2024 2024-04-24T22:47:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:47:07 2024 2024-04-24T22:47:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:47:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:47:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: ae301ebb-cfbe-4565-b97c-7f626fd5375e 2024-04-24T22:47:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:47:08.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:47:08.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_49 2024-04-24T22:47:08.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:47:08.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:47:08.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T22:47:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:47:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=5 2024-04-24T22:47:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:47:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:47:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 45716480 bytes 2024-04-24T22:47:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 82512896 bytes 2024-04-24T22:47:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 142418944 bytes 2024-04-24T22:47:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 181173248 bytes 2024-04-24T22:47:09.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 213024768 bytes 2024-04-24T22:47:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 324561920 bytes 2024-04-24T22:47:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 370596864 bytes 2024-04-24T22:47:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 423703552 bytes 2024-04-24T22:47:09.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 507080704 bytes 2024-04-24T22:47:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 612430848 bytes 2024-04-24T22:47:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 664184832 bytes 2024-04-24T22:47:10.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 668688384 bytes 2024-04-24T22:47:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 716614656 bytes 2024-04-24T22:47:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 738475008 bytes 2024-04-24T22:47:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 792120320 bytes 2024-04-24T22:47:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 799974400 bytes 2024-04-24T22:47:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 812252160 bytes 2024-04-24T22:47:11.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 840415232 bytes 2024-04-24T22:47:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 854820864 bytes 2024-04-24T22:47:11.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 930902016 bytes 2024-04-24T22:47:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 965636096 bytes 2024-04-24T22:47:11.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 988987392 bytes 2024-04-24T22:47:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1025443840 bytes 2024-04-24T22:47:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1072188416 bytes 2024-04-24T22:47:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1120345088 bytes 2024-04-24T22:47:12.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1157799936 bytes 2024-04-24T22:47:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1217953792 bytes 2024-04-24T22:47:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1242158080 bytes 2024-04-24T22:47:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1259600896 bytes 2024-04-24T22:47:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1334777856 bytes 2024-04-24T22:47:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1374156800 bytes 2024-04-24T22:47:13.012 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T22:47:13.010+0000 7f05853c0700 -1 reset not still connected to 0x5619ad952dd0 2024-04-24T22:47:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1391890432 bytes 2024-04-24T22:47:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1460225024 bytes 2024-04-24T22:47:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:47:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:47:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:47:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:47:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:47:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_50 2024-04-24T22:47:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2024-04-24T22:47:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:47:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:47:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:47:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:47:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T22:47:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:47:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:47:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n snapshot ']' 2024-04-24T22:47:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:47:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_50 128M 2024-04-24T22:47:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:47:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:47:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:47:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:47:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T22:47:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:47:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:47:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 128M ']' 2024-04-24T22:47:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=128M 2024-04-24T22:47:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:47:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_50 2024-04-24T22:47:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' image = image ']' 2024-04-24T22:47:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2024-04-24T22:47:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:47:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:47:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T22:47:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:47:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2024-04-24T22:47:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:47:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2024-04-24T22:47:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_50': 2024-04-24T22:47:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:47:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:47:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:47:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 14e1a7a72b3d 2024-04-24T22:47:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.14e1a7a72b3d 2024-04-24T22:47:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:47:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:47:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:47:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:47:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:47:13 2024 2024-04-24T22:47:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:47:13 2024 2024-04-24T22:47:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:47:13 2024 2024-04-24T22:47:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:47:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:47:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 19c08b8e-0c4b-4471-adf6-b6a40dd52550 2024-04-24T22:47:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:47:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n '' ']' 2024-04-24T22:47:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_50 2024-04-24T22:47:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:47:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:47:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T22:47:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:47:14.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=12 2024-04-24T22:47:14.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:47:14.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:47:14.424 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7986882832395463) > weight (0.25) 2024-04-24T22:47:14.424 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5235840640134495) > weight (0.25) 2024-04-24T22:47:14.424 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-04-24T22:47:14.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-04-24T22:47:14.425 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-04-24T22:47:14.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T22:47:14.424+0000 7f0fc4bc0700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 56082) UID: 1000 2024-04-24T22:47:14.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 63771648 bytes 2024-04-24T22:47:14.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:47:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 81087488 bytes 2024-04-24T22:47:14.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 97758208 bytes 2024-04-24T22:47:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 128009216 bytes 2024-04-24T22:47:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 165271552 bytes 2024-04-24T22:47:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 191773696 bytes 2024-04-24T22:47:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 208026624 bytes 2024-04-24T22:47:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 244068352 bytes 2024-04-24T22:47:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 335025152 bytes 2024-04-24T22:47:15.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 374424576 bytes 2024-04-24T22:47:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 413791232 bytes 2024-04-24T22:47:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 472501248 bytes 2024-04-24T22:47:16.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 512390144 bytes 2024-04-24T22:47:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 602806272 bytes 2024-04-24T22:47:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 682292224 bytes 2024-04-24T22:47:16.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 762610688 bytes 2024-04-24T22:47:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 790891520 bytes 2024-04-24T22:47:17.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 791513088 bytes 2024-04-24T22:47:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 821079040 bytes 2024-04-24T22:47:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 891940864 bytes 2024-04-24T22:47:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 925625344 bytes 2024-04-24T22:47:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 988234752 bytes 2024-04-24T22:47:17.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1033638912 bytes 2024-04-24T22:47:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1069679616 bytes 2024-04-24T22:47:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1143209984 bytes 2024-04-24T22:47:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1152503808 bytes 2024-04-24T22:47:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1170375680 bytes 2024-04-24T22:47:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1170981888 bytes 2024-04-24T22:47:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1193494528 bytes 2024-04-24T22:47:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1214593024 bytes 2024-04-24T22:47:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1252143104 bytes 2024-04-24T22:47:18.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1279954944 bytes 2024-04-24T22:47:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1339515904 bytes 2024-04-24T22:47:19.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1382974464 bytes 2024-04-24T22:47:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1421627392 bytes 2024-04-24T22:47:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1462482944 bytes 2024-04-24T22:47:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1492162560 bytes 2024-04-24T22:47:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1547634688 bytes 2024-04-24T22:47:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1570264064 bytes 2024-04-24T22:47:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1607968768 bytes 2024-04-24T22:47:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1687738368 bytes 2024-04-24T22:47:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1710394368 bytes 2024-04-24T22:47:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1748600832 bytes 2024-04-24T22:47:20.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1825610752 bytes 2024-04-24T22:47:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1892449280 bytes 2024-04-24T22:47:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1958267904 bytes 2024-04-24T22:47:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 2030014464 bytes 2024-04-24T22:47:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2102458368 bytes 2024-04-24T22:47:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2171842560 bytes 2024-04-24T22:47:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2230093824 bytes 2024-04-24T22:47:21.752 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T22:47:21.750+0000 7f05853c0700 -1 reset not still connected to 0x5619adfa2270 2024-04-24T22:47:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2265924608 bytes 2024-04-24T22:47:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2284481536 bytes 2024-04-24T22:47:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2338830336 bytes 2024-04-24T22:47:22.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2360755200 bytes 2024-04-24T22:47:22.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2404813824 bytes 2024-04-24T22:47:22.426 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-04-24T22:47:22.426 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:47:22.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:47:22.426 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-04-24T22:47:22.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-04-24T22:47:22.427 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-04-24T22:47:22.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-04-24T22:47:22.432 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2024-04-24T22:47:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2440467456 bytes 2024-04-24T22:47:22.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2510235648 bytes 2024-04-24T22:47:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2584648704 bytes 2024-04-24T22:47:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2587596800 bytes 2024-04-24T22:47:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2635658240 bytes 2024-04-24T22:47:23.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2674259968 bytes 2024-04-24T22:47:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2709523456 bytes 2024-04-24T22:47:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2715258880 bytes 2024-04-24T22:47:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2736228352 bytes 2024-04-24T22:47:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2754589696 bytes 2024-04-24T22:47:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2795366400 bytes 2024-04-24T22:47:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2811198464 bytes 2024-04-24T22:47:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2848826368 bytes 2024-04-24T22:47:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2887901184 bytes 2024-04-24T22:47:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 2910092288 bytes 2024-04-24T22:47:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 2965191680 bytes 2024-04-24T22:47:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 3059027968 bytes 2024-04-24T22:47:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 3068338176 bytes 2024-04-24T22:47:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 3133631488 bytes 2024-04-24T22:47:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 3227804672 bytes 2024-04-24T22:47:25.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 3269232640 bytes 2024-04-24T22:47:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 3365232640 bytes 2024-04-24T22:47:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 3476624384 bytes 2024-04-24T22:47:26.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7900 IOs, 3529268224 bytes 2024-04-24T22:47:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8000 IOs, 3567127552 bytes 2024-04-24T22:47:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8100 IOs, 3591493632 bytes 2024-04-24T22:47:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:47:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:47:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:47:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:47:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-04-24T22:47:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:47:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:47:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_count=50 2024-04-24T22:47:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:47:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local count 2024-04-24T22:47:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_count=0 2024-04-24T22:47:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:47:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 40 2024-04-24T22:47:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in `seq 1 40` 2024-04-24T22:47:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 1 -ne 1 2024-04-24T22:47:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-04-24T22:47:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep 'images: ' 2024-04-24T22:47:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut '-d ' -f 2 2024-04-24T22:47:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: failed to query services: (13) Permission denied 2024-04-24T22:47:26.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ count=50 2024-04-24T22:47:26.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 50 = 50 2024-04-24T22:47:26.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:47:26.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-04-24T22:47:26.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:47:26.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:47:26.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:47:26.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state 2024-04-24T22:47:26.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 40 2024-04-24T22:47:26.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in `seq 1 40` 2024-04-24T22:47:26.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 1 -ne 1 2024-04-24T22:47:26.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-04-24T22:47:26.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep 'image health:' 2024-04-24T22:47:26.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut '-d ' -f 3 2024-04-24T22:47:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: failed to query services: (13) Permission denied 2024-04-24T22:47:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=OK 2024-04-24T22:47:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test OK = ERROR 2024-04-24T22:47:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test OK = OK 2024-04-24T22:47:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:47:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-04-24T22:47:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:47:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:47:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_count=50 2024-04-24T22:47:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:47:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local count 2024-04-24T22:47:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_count=0 2024-04-24T22:47:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:47:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 40 2024-04-24T22:47:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in `seq 1 40` 2024-04-24T22:47:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 1 -ne 1 2024-04-24T22:47:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-04-24T22:47:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep 'images: ' 2024-04-24T22:47:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut '-d ' -f 2 2024-04-24T22:47:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: failed to query services: (13) Permission denied 2024-04-24T22:47:26.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ count=50 2024-04-24T22:47:26.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 50 = 50 2024-04-24T22:47:26.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:47:26.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-04-24T22:47:26.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:47:26.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:47:26.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:47:26.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state 2024-04-24T22:47:26.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 40 2024-04-24T22:47:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in `seq 1 40` 2024-04-24T22:47:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 1 -ne 1 2024-04-24T22:47:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-04-24T22:47:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep 'image health:' 2024-04-24T22:47:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut '-d ' -f 3 2024-04-24T22:47:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: failed to query services: (13) Permission denied 2024-04-24T22:47:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=WARNING 2024-04-24T22:47:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test WARNING = ERROR 2024-04-24T22:47:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test WARNING = OK 2024-04-24T22:47:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in `seq 1 40` 2024-04-24T22:47:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 2 -ne 1 2024-04-24T22:47:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 30 2024-04-24T22:47:51.652 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7095887860519913) > weight (0.25) 2024-04-24T22:47:51.653 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.2721793777693313) > weight (0.25) 2024-04-24T22:47:51.653 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-04-24T22:47:51.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-04-24T22:47:51.653 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-04-24T22:47:51.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T22:47:51.653+0000 7f888cdbc700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 58035) UID: 1000 2024-04-24T22:47:51.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:47:54.654 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-04-24T22:47:54.654 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:47:54.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:47:54.655 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-04-24T22:47:54.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-24T22:47:54.655 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-24T22:47:54.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-24T22:47:54.658 INFO:tasks.rbd_mirror_thrash:waiting for 79 secs before thrashing 2024-04-24T22:47:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-04-24T22:47:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep 'image health:' 2024-04-24T22:47:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut '-d ' -f 3 2024-04-24T22:47:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: failed to query services: (13) Permission denied 2024-04-24T22:47:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=OK 2024-04-24T22:47:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test OK = ERROR 2024-04-24T22:47:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test OK = OK 2024-04-24T22:47:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:47:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ testlog 'TEST: compare many images' 2024-04-24T22:47:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ tee -a /tmp/tmp.DCvHjwkiaM/rbd-mirror.test.log 2024-04-24T22:47:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ date '+%F %T' 2024-04-24T22:47:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo 2024-04-24 22:47:56 TEST: compare many images 2024-04-24T22:47:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:2024-04-24 22:47:56 TEST: compare many images 2024-04-24T22:47:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 50 2024-04-24T22:47:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:47:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_1 2024-04-24T22:47:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_1 snap 2024-04-24T22:47:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:47:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:47:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:47:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:47:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:47:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-04-24T22:47:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:47:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:47:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:47:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-04-24T22:47:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:47:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:47:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:47:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:47:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:47:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:47:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:47:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-04-24T22:47:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:47:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:47:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:47:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:47:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:47:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:47:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-04-24T22:47:59.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:47:59.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:47:59.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:47:59.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:47:59.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:47:59.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:47:59.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:47:59.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:47:59.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:47:59.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:47:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:47:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-04-24T22:47:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:47:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:47:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:47:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:47:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:47:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:47:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:47:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:47:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:47:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:47:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:47:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:47:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:47:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:47:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:47:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:47:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:47:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:47:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:47:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:47:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:47:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:47:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:47:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:47:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:47:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:47:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:47:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:47:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:47:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:47:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:47:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:47:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:47:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-04-24T22:47:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-04-24T22:47:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:47:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:47:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:47:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2024-04-24T22:47:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:47:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:47:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:47:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:47:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:47:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:47:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:47:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:47:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:47:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:47:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:47:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:47:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:47:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:47:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:47:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:47:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:47:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:47:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:47:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:47:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:47:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:47:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:47:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:47:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:47:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:47:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:47:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:47:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:47:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:47:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:47:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:47:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:47:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2024-04-24T22:47:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-04-24T22:47:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:47:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:47:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:47:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_1 2024-04-24T22:47:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:47:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:47:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:47:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:47:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:47:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:47:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:47:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:47:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:47:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:47:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:47:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:47:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:47:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:47:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:47:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:47:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:47:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:47:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:47:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:47:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:47:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:47:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:47:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:47:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:47:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:47:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:47:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:47:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:47:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:47:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:47:59.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:47:59.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:47:59.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_1 2024-04-24T22:47:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-04-24T22:47:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:47:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:47:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:47:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_1 2024-04-24T22:47:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:47:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:47:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:47:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:47:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:47:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:47:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:47:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:47:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:47:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:47:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:47:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:47:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:47:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:47:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:47:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:47:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:47:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:47:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:47:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:47:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:47:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:47:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:47:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:47:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:47:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:47:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:47:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:47:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:47:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:47:59.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:47:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:47:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:47:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_1 2024-04-24T22:47:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:47:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:47:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:47:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:47:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:47:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:47:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:47:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:47:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2024-04-24T22:47:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:47:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:47:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:47:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:47:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:47:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:47:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2024-04-24T22:47:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:47:59.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:47:59.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:47:59.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:47:59.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_1.status 2024-04-24T22:47:59.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_1.status 2024-04-24T22:47:59.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:47:59.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_1.status 2024-04-24T22:47:59.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_1.status 2024-04-24T22:47:59.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_1.status 2024-04-24T22:47:59.750 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:47:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_1.status 2024-04-24T22:47:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2024-04-24T22:47:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:47:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:47:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:47:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2024-04-24T22:48:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 119 2024-04-24T22:48:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_1.status 2024-04-24T22:48:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:48:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_1.status 2024-04-24T22:48:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2024-04-24T22:48:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:48:00.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=119 2024-04-24T22:48:00.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:00.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:00.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:48:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_1.status 2024-04-24T22:48:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:48:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_1.status 2024-04-24T22:48:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-04-24T22:48:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:00.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:00.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:48:00.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:48:00.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:48:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:48:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 119 = '' 2024-04-24T22:48:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:48:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_1.status 2024-04-24T22:48:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:48:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_1.status 2024-04-24T22:48:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-04-24T22:48:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:01.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:01.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:48:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:48:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:48:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:48:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 119 = '' 2024-04-24T22:48:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:48:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_1.status 2024-04-24T22:48:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:48:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_1.status 2024-04-24T22:48:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-04-24T22:48:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:02.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=119 2024-04-24T22:48:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 119 = 119 2024-04-24T22:48:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-04-24T22:48:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:48:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:48:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:48:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:48:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-04-24T22:48:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_1': 2024-04-24T22:48:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:48:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:48:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:48:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc51671464 2024-04-24T22:48:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc51671464 2024-04-24T22:48:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:48:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:48:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:48:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:48:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:39:28 2024 2024-04-24T22:48:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:39:28 2024 2024-04-24T22:48:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:39:28 2024 2024-04-24T22:48:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:48:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:48:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:48:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: a0aa7e18-e06a-4e3c-b4d7-6bdccb4d22a8 2024-04-24T22:48:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:48:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_1 snap 2024-04-24T22:48:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:48:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:48:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_1.export 2024-04-24T22:48:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_1.export 2024-04-24T22:48:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_1.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_1.export 2024-04-24T22:48:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_1.export 2024-04-24T22:48:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:48:03.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_1.export 2024-04-24T22:48:03.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:48:03.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_1.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_1.export 2024-04-24T22:48:03.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_1.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_1.export 2024-04-24T22:48:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:48:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_2 2024-04-24T22:48:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_2 snap 2024-04-24T22:48:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:48:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:48:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-04-24T22:48:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:48:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-04-24T22:48:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:48:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:48:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:48:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:48:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:48:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-04-24T22:48:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:48:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:48:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:48:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:48:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-04-24T22:48:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:48:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:48:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:48:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:48:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:48:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:48:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:48:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:48:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:48:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-04-24T22:48:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:48:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:48:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:48:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:48:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:48:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:48:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:48:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:48:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:48:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:48:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:48:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:06.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:06.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:48:06.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:48:06.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:06.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:06.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:48:06.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:48:06.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:48:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-04-24T22:48:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-04-24T22:48:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2024-04-24T22:48:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:48:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:48:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:48:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:48:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:48:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:48:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:48:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:48:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:48:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:48:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:48:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:48:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:48:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:48:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:48:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:48:06.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:06.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:06.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:06.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2024-04-24T22:48:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-04-24T22:48:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:06.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:06.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2024-04-24T22:48:06.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:48:06.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:06.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:48:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:48:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:48:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:48:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:48:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:48:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:48:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:48:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:48:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:48:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:48:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:48:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:48:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:48:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:48:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2024-04-24T22:48:06.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-04-24T22:48:06.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:06.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_2 2024-04-24T22:48:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:48:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:48:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:48:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:48:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:48:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:48:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:48:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:48:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:48:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:48:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:48:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:48:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:48:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:06.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:06.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:48:06.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:48:06.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:48:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_2 2024-04-24T22:48:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:48:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:48:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:48:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:48:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:48:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:48:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:48:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2024-04-24T22:48:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:48:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:48:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:48:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:48:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2024-04-24T22:48:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:48:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:48:06.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_2.status 2024-04-24T22:48:06.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_2.status 2024-04-24T22:48:06.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:48:06.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_2.status 2024-04-24T22:48:06.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_2.status 2024-04-24T22:48:06.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_2.status 2024-04-24T22:48:06.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:48:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_2.status 2024-04-24T22:48:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2024-04-24T22:48:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:48:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2024-04-24T22:48:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 121 2024-04-24T22:48:07.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_2.status 2024-04-24T22:48:07.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:07.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:07.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:48:07.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_2.status 2024-04-24T22:48:07.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2024-04-24T22:48:07.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:48:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=121 2024-04-24T22:48:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:48:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_2.status 2024-04-24T22:48:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:48:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_2.status 2024-04-24T22:48:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-04-24T22:48:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:48:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:48:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:48:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:48:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 121 = '' 2024-04-24T22:48:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:48:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_2.status 2024-04-24T22:48:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:48:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_2.status 2024-04-24T22:48:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-04-24T22:48:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:48:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:48:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:48:08.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:48:08.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 121 = '' 2024-04-24T22:48:08.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:08.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:48:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_2.status 2024-04-24T22:48:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:48:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_2.status 2024-04-24T22:48:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-04-24T22:48:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=121 2024-04-24T22:48:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 121 = 121 2024-04-24T22:48:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-04-24T22:48:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:48:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:48:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:48:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:48:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-04-24T22:48:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_2': 2024-04-24T22:48:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:48:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:48:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:48:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc8916c341 2024-04-24T22:48:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc8916c341 2024-04-24T22:48:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:48:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:48:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:48:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:48:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:39:34 2024 2024-04-24T22:48:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:39:34 2024 2024-04-24T22:48:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:39:34 2024 2024-04-24T22:48:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:48:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:48:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:48:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 9d47d6f5-f040-4915-aac4-a0dd1e04cc47 2024-04-24T22:48:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:48:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_2 snap 2024-04-24T22:48:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:48:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:48:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_2.export 2024-04-24T22:48:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_2.export 2024-04-24T22:48:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_2.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_2.export 2024-04-24T22:48:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_2.export 2024-04-24T22:48:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:48:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_2.export 2024-04-24T22:48:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:48:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_2.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_2.export 2024-04-24T22:48:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_2.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_2.export 2024-04-24T22:48:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:48:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_3 2024-04-24T22:48:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_3 snap 2024-04-24T22:48:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:48:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:48:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-04-24T22:48:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:48:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-04-24T22:48:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:48:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:48:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:48:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:48:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:48:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-04-24T22:48:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:48:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:48:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:48:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:48:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-04-24T22:48:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:48:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:48:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:48:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:48:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:48:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:48:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:48:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:48:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:48:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-04-24T22:48:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:48:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:13.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:13.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:48:13.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:48:13.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:13.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:13.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:13.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:48:13.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:48:13.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:13.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:13.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:48:13.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:48:13.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:48:13.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:48:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:48:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:48:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:48:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:48:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:48:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:48:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:48:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-04-24T22:48:13.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2024-04-24T22:48:13.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:13.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2024-04-24T22:48:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:48:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:48:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:48:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:48:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:48:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:48:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:48:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:48:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:48:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:48:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:48:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:48:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:48:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:48:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:48:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:48:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2024-04-24T22:48:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2024-04-24T22:48:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_3 2024-04-24T22:48:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:48:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:48:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:48:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:48:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:48:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:48:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:48:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:48:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:48:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:48:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:48:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:48:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:48:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:48:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:48:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:48:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_3 2024-04-24T22:48:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2024-04-24T22:48:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_3 2024-04-24T22:48:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:48:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:48:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:48:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:48:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:48:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:48:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:48:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:48:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:48:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:48:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:48:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:48:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:48:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:48:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:48:13.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:48:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_3 2024-04-24T22:48:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:48:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:48:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:48:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:48:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:48:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:48:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:48:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2024-04-24T22:48:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:48:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:48:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:48:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:48:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2024-04-24T22:48:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:48:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:48:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_3.status 2024-04-24T22:48:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_3.status 2024-04-24T22:48:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:48:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_3.status 2024-04-24T22:48:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_3.status 2024-04-24T22:48:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_3.status 2024-04-24T22:48:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:48:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_3.status 2024-04-24T22:48:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2024-04-24T22:48:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:48:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2024-04-24T22:48:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 123 2024-04-24T22:48:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_3.status 2024-04-24T22:48:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:48:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_3.status 2024-04-24T22:48:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2024-04-24T22:48:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:48:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=123 2024-04-24T22:48:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:48:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_3.status 2024-04-24T22:48:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:48:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_3.status 2024-04-24T22:48:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-04-24T22:48:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:48:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:48:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:48:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:48:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 123 = '' 2024-04-24T22:48:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:48:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_3.status 2024-04-24T22:48:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:48:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_3.status 2024-04-24T22:48:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-04-24T22:48:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:48:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:48:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:48:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:48:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 123 = '' 2024-04-24T22:48:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:48:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_3.status 2024-04-24T22:48:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:48:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_3.status 2024-04-24T22:48:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-04-24T22:48:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=123 2024-04-24T22:48:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 123 = 123 2024-04-24T22:48:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-04-24T22:48:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:48:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:48:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:48:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:48:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-04-24T22:48:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_3': 2024-04-24T22:48:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:48:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:48:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:48:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dcbf790c4e 2024-04-24T22:48:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dcbf790c4e 2024-04-24T22:48:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:48:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:48:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:48:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:48:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:39:47 2024 2024-04-24T22:48:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:39:47 2024 2024-04-24T22:48:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:39:47 2024 2024-04-24T22:48:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:48:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:48:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:48:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: e799308a-a04d-4e8c-b6d0-1485ec8486a2 2024-04-24T22:48:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:48:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_3 snap 2024-04-24T22:48:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:48:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:48:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_3.export 2024-04-24T22:48:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_3.export 2024-04-24T22:48:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_3.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_3.export 2024-04-24T22:48:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_3.export 2024-04-24T22:48:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:48:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_3.export 2024-04-24T22:48:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:48:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_3.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_3.export 2024-04-24T22:48:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_3.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_3.export 2024-04-24T22:48:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:48:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_4 2024-04-24T22:48:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_4 snap 2024-04-24T22:48:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:48:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:48:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-04-24T22:48:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:48:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-04-24T22:48:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:48:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:48:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:48:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:48:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:48:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-04-24T22:48:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:48:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:48:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:48:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:48:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-04-24T22:48:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:48:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:48:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:48:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:48:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:48:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:48:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:48:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:48:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:48:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-04-24T22:48:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:48:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:48:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:48:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:48:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:48:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:48:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:48:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:48:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:48:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:48:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:48:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:48:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:48:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:48:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:48:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:48:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-04-24T22:48:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2024-04-24T22:48:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2024-04-24T22:48:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:48:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:48:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:48:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:48:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:48:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:48:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:48:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:48:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:48:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:48:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:48:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:48:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:48:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:48:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:48:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:48:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2024-04-24T22:48:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2024-04-24T22:48:20.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:20.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2024-04-24T22:48:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:48:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:48:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:48:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:48:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:48:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:48:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:48:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:48:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:48:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:48:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:48:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:48:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:48:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:48:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:48:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:48:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2024-04-24T22:48:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2024-04-24T22:48:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_4 2024-04-24T22:48:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:48:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:48:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:48:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:48:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:48:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:48:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:48:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:48:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:48:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:48:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:48:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:48:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:48:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:48:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:48:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:48:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_4 2024-04-24T22:48:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:48:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:48:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:48:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:48:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:48:20.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:48:20.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:48:20.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:20.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2024-04-24T22:48:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:48:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:48:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:48:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:48:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2024-04-24T22:48:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:48:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:48:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_4.status 2024-04-24T22:48:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_4.status 2024-04-24T22:48:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:48:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_4.status 2024-04-24T22:48:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_4.status 2024-04-24T22:48:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_4.status 2024-04-24T22:48:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:48:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_4.status 2024-04-24T22:48:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2024-04-24T22:48:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:48:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2024-04-24T22:48:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 125 2024-04-24T22:48:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_4.status 2024-04-24T22:48:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:48:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_4.status 2024-04-24T22:48:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2024-04-24T22:48:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:48:21.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=125 2024-04-24T22:48:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:48:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_4.status 2024-04-24T22:48:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:48:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_4.status 2024-04-24T22:48:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-04-24T22:48:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:21.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:21.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:48:21.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:48:21.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:48:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:48:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 125 = '' 2024-04-24T22:48:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:48:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_4.status 2024-04-24T22:48:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:48:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_4.status 2024-04-24T22:48:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-04-24T22:48:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:22.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:48:22.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:48:22.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:48:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:48:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 125 = '' 2024-04-24T22:48:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:48:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_4.status 2024-04-24T22:48:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:48:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_4.status 2024-04-24T22:48:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-04-24T22:48:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=125 2024-04-24T22:48:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 125 = 125 2024-04-24T22:48:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-04-24T22:48:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:48:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:48:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:48:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:48:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-04-24T22:48:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_4': 2024-04-24T22:48:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:48:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:48:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:48:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dcfa07b92a 2024-04-24T22:48:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dcfa07b92a 2024-04-24T22:48:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:48:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:48:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:48:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:48:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:39:54 2024 2024-04-24T22:48:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:39:54 2024 2024-04-24T22:48:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:39:54 2024 2024-04-24T22:48:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:48:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:48:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:48:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 455d42c6-ae74-4f65-ae3f-0405438be886 2024-04-24T22:48:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:48:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_4 snap 2024-04-24T22:48:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:48:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:48:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_4.export 2024-04-24T22:48:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_4.export 2024-04-24T22:48:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_4.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_4.export 2024-04-24T22:48:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_4.export 2024-04-24T22:48:24.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:48:24.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_4.export 2024-04-24T22:48:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:48:24.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_4.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_4.export 2024-04-24T22:48:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_4.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_4.export 2024-04-24T22:48:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:48:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_5 2024-04-24T22:48:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_5 snap 2024-04-24T22:48:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:48:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:48:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-04-24T22:48:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:48:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-04-24T22:48:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:48:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:48:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:48:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:48:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:48:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-04-24T22:48:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:48:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:48:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:48:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:48:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-04-24T22:48:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:48:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:48:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:48:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:48:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:48:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:48:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:48:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:48:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:48:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-04-24T22:48:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:48:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:48:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:48:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:48:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:48:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:48:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:48:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:48:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:48:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:48:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:48:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:48:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:48:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:48:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:48:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:48:27.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:27.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-04-24T22:48:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-04-24T22:48:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-04-24T22:48:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:48:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:48:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:48:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:48:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:48:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:48:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:48:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:48:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:48:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:48:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:48:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:48:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:48:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:48:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:48:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:48:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-04-24T22:48:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-04-24T22:48:27.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:27.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:27.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:27.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2024-04-24T22:48:27.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:48:27.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:27.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:27.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:48:27.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:48:27.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:27.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:27.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:27.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:48:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:48:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:48:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:48:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:48:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:48:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:48:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:48:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:27.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:48:27.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:48:27.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:27.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:27.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:48:27.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:48:27.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:48:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2024-04-24T22:48:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-04-24T22:48:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2024-04-24T22:48:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:48:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:48:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:48:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:48:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:48:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:48:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:48:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:48:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:48:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:48:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:48:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:48:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:48:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:48:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:48:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:48:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2024-04-24T22:48:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:48:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:48:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:48:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:48:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:48:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:48:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:48:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2024-04-24T22:48:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:48:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:48:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:48:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:48:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2024-04-24T22:48:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:48:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:48:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_5.status 2024-04-24T22:48:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_5.status 2024-04-24T22:48:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:48:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_5.status 2024-04-24T22:48:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_5.status 2024-04-24T22:48:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_5.status 2024-04-24T22:48:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:48:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_5.status 2024-04-24T22:48:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2024-04-24T22:48:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:48:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2024-04-24T22:48:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 127 2024-04-24T22:48:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_5.status 2024-04-24T22:48:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:48:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_5.status 2024-04-24T22:48:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2024-04-24T22:48:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:48:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=127 2024-04-24T22:48:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:48:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_5.status 2024-04-24T22:48:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:48:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_5.status 2024-04-24T22:48:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-04-24T22:48:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:48:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:48:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:48:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:48:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 127 = '' 2024-04-24T22:48:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:48:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_5.status 2024-04-24T22:48:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:48:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_5.status 2024-04-24T22:48:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-04-24T22:48:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:48:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:48:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:48:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:48:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 127 = '' 2024-04-24T22:48:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:48:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_5.status 2024-04-24T22:48:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:48:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_5.status 2024-04-24T22:48:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-04-24T22:48:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=127 2024-04-24T22:48:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 127 = 127 2024-04-24T22:48:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-04-24T22:48:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:48:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:48:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:48:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:48:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-04-24T22:48:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_5': 2024-04-24T22:48:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:48:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:48:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:48:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dcbaa3ef21 2024-04-24T22:48:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dcbaa3ef21 2024-04-24T22:48:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:48:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:48:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:48:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:48:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:40:00 2024 2024-04-24T22:48:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:40:00 2024 2024-04-24T22:48:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:40:00 2024 2024-04-24T22:48:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:48:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:48:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:48:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 6fb19b55-9e60-4b51-937c-bfff39ac7c68 2024-04-24T22:48:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:48:30.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:30.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_5 snap 2024-04-24T22:48:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:48:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:48:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_5.export 2024-04-24T22:48:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_5.export 2024-04-24T22:48:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_5.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_5.export 2024-04-24T22:48:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_5.export 2024-04-24T22:48:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:48:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_5.export 2024-04-24T22:48:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:48:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_5.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_5.export 2024-04-24T22:48:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_5.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_5.export 2024-04-24T22:48:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:48:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_6 2024-04-24T22:48:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_6 snap 2024-04-24T22:48:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T22:48:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:48:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-04-24T22:48:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T22:48:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-04-24T22:48:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T22:48:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:48:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:48:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:48:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:48:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-04-24T22:48:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T22:48:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:48:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:48:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:48:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-04-24T22:48:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:48:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:48:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:48:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:48:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:48:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:48:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:48:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:48:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:48:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-04-24T22:48:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:48:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:48:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:48:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:48:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:48:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:48:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:48:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:48:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:48:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:48:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:48:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:48:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:48:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:48:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:48:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:48:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-04-24T22:48:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-04-24T22:48:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2024-04-24T22:48:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:48:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:48:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:48:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:48:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:48:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:48:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:48:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:48:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:48:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:48:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:48:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:48:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:48:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:48:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:48:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:48:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2024-04-24T22:48:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-04-24T22:48:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_6 2024-04-24T22:48:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:48:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:48:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:48:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:48:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:48:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:48:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:48:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:48:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:48:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:48:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:48:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:48:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:48:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:48:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:48:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:48:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_6 2024-04-24T22:48:34.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-04-24T22:48:34.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:34.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_6 2024-04-24T22:48:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:48:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:48:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:48:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:48:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:48:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:48:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:48:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:48:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:48:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:48:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:48:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:48:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:48:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:48:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:48:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:48:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_6 2024-04-24T22:48:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:48:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:48:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:48:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:48:34.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:48:34.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:48:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:48:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2024-04-24T22:48:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:48:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T22:48:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:48:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:48:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2024-04-24T22:48:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:48:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T22:48:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_6.status 2024-04-24T22:48:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_6.status 2024-04-24T22:48:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:48:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_6.status 2024-04-24T22:48:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_6.status 2024-04-24T22:48:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_6.status 2024-04-24T22:48:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:48:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_6.status 2024-04-24T22:48:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2024-04-24T22:48:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T22:48:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2024-04-24T22:48:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 129 2024-04-24T22:48:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_6.status 2024-04-24T22:48:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T22:48:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_6.status 2024-04-24T22:48:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2024-04-24T22:48:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:35.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:48:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=129 2024-04-24T22:48:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:48:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_6.status 2024-04-24T22:48:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T22:48:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_6.status 2024-04-24T22:48:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-04-24T22:48:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:48:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:48:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:48:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:48:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 129 = '' 2024-04-24T22:48:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:48:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_6.status 2024-04-24T22:48:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T22:48:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_6.status 2024-04-24T22:48:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-04-24T22:48:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:48:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:48:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:48:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:48:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 129 = '' 2024-04-24T22:48:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:48:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_6.status 2024-04-24T22:48:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T22:48:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_6.status 2024-04-24T22:48:36.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-04-24T22:48:36.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=129 2024-04-24T22:48:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 129 = 129 2024-04-24T22:48:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-04-24T22:48:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T22:48:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:48:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:48:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:48:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-04-24T22:48:37.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_6': 2024-04-24T22:48:37.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:48:37.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:48:37.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:48:37.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc9ef45426 2024-04-24T22:48:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc9ef45426 2024-04-24T22:48:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:48:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:48:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:48:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:48:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:40:13 2024 2024-04-24T22:48:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:40:13 2024 2024-04-24T22:48:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:40:13 2024 2024-04-24T22:48:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:48:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:48:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:48:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 059ce48e-b70a-434e-9dd3-e955c7a954da 2024-04-24T22:48:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:48:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_6 snap 2024-04-24T22:48:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T22:48:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:48:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_6.export 2024-04-24T22:48:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_6.export 2024-04-24T22:48:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_6.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_6.export 2024-04-24T22:48:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_6.export 2024-04-24T22:48:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:48:38.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_6.export 2024-04-24T22:48:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:48:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_6.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_6.export 2024-04-24T22:48:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_6.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_6.export 2024-04-24T22:48:38.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:38.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:48:38.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_7 2024-04-24T22:48:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_7 snap 2024-04-24T22:48:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T22:48:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:48:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-04-24T22:48:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T22:48:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-04-24T22:48:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T22:48:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:48:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:48:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:48:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:48:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-04-24T22:48:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:41.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T22:48:41.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:48:41.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:48:41.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:48:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-04-24T22:48:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:48:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:48:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:48:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:48:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:48:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:48:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:48:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:48:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:48:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-04-24T22:48:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:48:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:48:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:48:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:48:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:48:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:48:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:48:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:48:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:48:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:48:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:48:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:48:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:48:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:48:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:48:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:48:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-04-24T22:48:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-04-24T22:48:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:41.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2024-04-24T22:48:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:48:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:48:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:48:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:48:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:48:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:48:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:48:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:48:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:48:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:48:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:48:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:48:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:48:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:48:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:48:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:48:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2024-04-24T22:48:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-04-24T22:48:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_7 2024-04-24T22:48:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:48:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:48:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:48:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:48:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:48:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:48:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:48:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:48:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:48:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:48:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:48:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:48:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:48:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:41.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:48:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:48:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:48:41.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:41.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:41.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:41.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_7 2024-04-24T22:48:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-04-24T22:48:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_7 2024-04-24T22:48:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:48:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:48:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:48:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:48:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:48:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:48:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:48:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:48:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:48:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:48:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:48:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:48:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:48:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:48:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:48:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:48:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_7 2024-04-24T22:48:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:48:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:48:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:48:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:48:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:48:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:48:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:48:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2024-04-24T22:48:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:48:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T22:48:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:48:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:48:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2024-04-24T22:48:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:48:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T22:48:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_7.status 2024-04-24T22:48:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_7.status 2024-04-24T22:48:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:48:41.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_7.status 2024-04-24T22:48:41.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_7.status 2024-04-24T22:48:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_7.status 2024-04-24T22:48:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:48:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_7.status 2024-04-24T22:48:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2024-04-24T22:48:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T22:48:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2024-04-24T22:48:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 131 2024-04-24T22:48:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_7.status 2024-04-24T22:48:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:42.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T22:48:42.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_7.status 2024-04-24T22:48:42.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2024-04-24T22:48:42.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:48:42.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=131 2024-04-24T22:48:42.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:42.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:42.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:48:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_7.status 2024-04-24T22:48:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T22:48:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_7.status 2024-04-24T22:48:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-04-24T22:48:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:48:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:48:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:48:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:48:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 131 = '' 2024-04-24T22:48:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:48:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_7.status 2024-04-24T22:48:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T22:48:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_7.status 2024-04-24T22:48:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-04-24T22:48:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:48:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:48:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:48:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:48:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 131 = '' 2024-04-24T22:48:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:48:43.626 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T22:48:43.616+0000 7fb1e9f85700 -1 reset not still connected to 0x5586bc9504e0 2024-04-24T22:48:43.626 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T22:48:43.620+0000 7fb1e9f85700 -1 reset not still connected to 0x5586bc951c70 2024-04-24T22:48:43.626 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T22:48:43.623+0000 7fb1e9f85700 -1 reset not still connected to 0x5586c13c11e0 2024-04-24T22:48:43.627 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T22:48:43.625+0000 7fb1e9f85700 -1 reset not still connected to 0x5586cf59ea90 2024-04-24T22:48:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_7.status 2024-04-24T22:48:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T22:48:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_7.status 2024-04-24T22:48:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-04-24T22:48:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=131 2024-04-24T22:48:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 131 = 131 2024-04-24T22:48:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-04-24T22:48:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T22:48:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:48:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:48:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:48:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-04-24T22:48:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_7': 2024-04-24T22:48:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:48:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:48:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:48:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dccd8e14e 2024-04-24T22:48:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dccd8e14e 2024-04-24T22:48:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:48:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:48:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:48:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:48:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:40:25 2024 2024-04-24T22:48:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:40:25 2024 2024-04-24T22:48:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:40:25 2024 2024-04-24T22:48:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:48:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:48:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:48:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 3cf64901-3ddb-4c2f-aca5-336fbdf69cfe 2024-04-24T22:48:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:48:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_7 snap 2024-04-24T22:48:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T22:48:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:48:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_7.export 2024-04-24T22:48:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_7.export 2024-04-24T22:48:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_7.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_7.export 2024-04-24T22:48:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_7.export 2024-04-24T22:48:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:48:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_7.export 2024-04-24T22:48:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:48:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_7.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_7.export 2024-04-24T22:48:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_7.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_7.export 2024-04-24T22:48:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:48:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_8 2024-04-24T22:48:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_8 snap 2024-04-24T22:48:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T22:48:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:48:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-04-24T22:48:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T22:48:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-04-24T22:48:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T22:48:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:48:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:48:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:48:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:48:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-04-24T22:48:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T22:48:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:48:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:48:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:48:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-04-24T22:48:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:48:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:48:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:48:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:48:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:48:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:48:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:48:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:48:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:48:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-04-24T22:48:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:48:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:48:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:48:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:48:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:48:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:48:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:48:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:48:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:48:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:48:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:48:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:48:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:48:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:48:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:48:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:48:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-04-24T22:48:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-24T22:48:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2024-04-24T22:48:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:48:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:48:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:48:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:48:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:48:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:48:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:48:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:48:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:48:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:48:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:48:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:48:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:48:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:48:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:48:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:48:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2024-04-24T22:48:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-24T22:48:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_8 2024-04-24T22:48:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:48:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:48:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:48:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:48:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:48:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:48:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:48:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:48:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:48:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:48:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:48:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:48:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:48:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:48:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:48:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:48:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_8 2024-04-24T22:48:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-04-24T22:48:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_8 2024-04-24T22:48:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:48:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:48:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:48:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:48:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:48:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:48:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:48:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:48:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:48:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:48:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:48:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:48:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:48:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:48:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:48:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:48:47.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:47.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:47.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:47.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_8 2024-04-24T22:48:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:48:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:48:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:48:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:48:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:48:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:48:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:48:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2024-04-24T22:48:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:48:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T22:48:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:48:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:48:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2024-04-24T22:48:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:48:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T22:48:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_8.status 2024-04-24T22:48:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_8.status 2024-04-24T22:48:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:48:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_8.status 2024-04-24T22:48:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_8.status 2024-04-24T22:48:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_8.status 2024-04-24T22:48:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:48:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_8.status 2024-04-24T22:48:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2024-04-24T22:48:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T22:48:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2024-04-24T22:48:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 133 2024-04-24T22:48:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_8.status 2024-04-24T22:48:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T22:48:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_8.status 2024-04-24T22:48:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2024-04-24T22:48:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:48:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=133 2024-04-24T22:48:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:48:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_8.status 2024-04-24T22:48:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T22:48:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_8.status 2024-04-24T22:48:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-04-24T22:48:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=75 2024-04-24T22:48:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 133 = 75 2024-04-24T22:48:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:48:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_8.status 2024-04-24T22:48:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T22:48:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_8.status 2024-04-24T22:48:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-04-24T22:48:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=75 2024-04-24T22:48:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 133 = 75 2024-04-24T22:48:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:49.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_8.status 2024-04-24T22:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T22:48:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_8.status 2024-04-24T22:48:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-04-24T22:48:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:48:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:48:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:48:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:48:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 133 = '' 2024-04-24T22:48:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T22:48:51.433 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-24T22:48:51.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-24T22:48:51.434 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-04-24T22:48:51.434 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-24T22:48:51.434 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:48:51.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:48:51.433+0000 7fb1cebf6700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 59356) UID: 1000 2024-04-24T22:48:51.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:48:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_8.status 2024-04-24T22:48:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T22:48:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_8.status 2024-04-24T22:48:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-04-24T22:48:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:51.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=133 2024-04-24T22:48:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 133 = 133 2024-04-24T22:48:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-04-24T22:48:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T22:48:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:48:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:48:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:48:52.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-04-24T22:48:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_8': 2024-04-24T22:48:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:48:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:48:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:48:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc43821aba 2024-04-24T22:48:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc43821aba 2024-04-24T22:48:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:48:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:48:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:48:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:48:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:40:32 2024 2024-04-24T22:48:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:40:32 2024 2024-04-24T22:48:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:40:32 2024 2024-04-24T22:48:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:48:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:48:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:48:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 8f5f4362-e6f1-477d-bce4-f3ab1a7caee3 2024-04-24T22:48:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:48:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_8 snap 2024-04-24T22:48:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T22:48:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:48:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_8.export 2024-04-24T22:48:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_8.export 2024-04-24T22:48:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_8.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_8.export 2024-04-24T22:48:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_8.export 2024-04-24T22:48:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:48:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_8.export 2024-04-24T22:48:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:48:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_8.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_8.export 2024-04-24T22:48:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_8.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_8.export 2024-04-24T22:48:53.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:53.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:48:53.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_9 2024-04-24T22:48:53.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_9 snap 2024-04-24T22:48:53.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:53.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:53.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T22:48:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:48:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-04-24T22:48:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T22:48:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-04-24T22:48:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T22:48:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:48:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:48:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:48:55.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:48:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-04-24T22:48:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T22:48:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:48:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:48:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:48:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-04-24T22:48:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:48:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:48:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:48:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:48:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:48:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:48:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:48:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:48:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:48:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-04-24T22:48:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:48:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:48:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:48:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:48:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:48:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:48:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:48:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:48:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:48:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:48:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:48:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:48:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:48:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:48:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:48:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:48:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:48:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-04-24T22:48:56.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-04-24T22:48:56.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:56.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:56.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:56.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2024-04-24T22:48:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:48:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:48:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:48:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:48:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:48:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:48:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:48:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:48:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:48:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:48:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:48:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:48:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:48:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:48:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:48:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:48:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:48:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2024-04-24T22:48:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-04-24T22:48:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_9 2024-04-24T22:48:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:48:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:48:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:48:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:48:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:48:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:48:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:48:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:48:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:48:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:48:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:48:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:48:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:48:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:48:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:48:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:48:56.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:48:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_9 2024-04-24T22:48:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-04-24T22:48:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:48:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:48:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:48:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_9 2024-04-24T22:48:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:48:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:48:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:48:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:48:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:48:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:48:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:48:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:48:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:48:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:48:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:48:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:48:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:48:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:48:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:48:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:48:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:48:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:48:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:48:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:48:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:48:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:48:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:48:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:48:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:48:56.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:48:56.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:48:56.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:48:56.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:48:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:48:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_9 2024-04-24T22:48:56.756 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:48:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:48:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:48:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:48:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:48:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:48:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:48:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2024-04-24T22:48:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:48:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T22:48:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:48:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:48:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2024-04-24T22:48:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:48:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T22:48:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_9.status 2024-04-24T22:48:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_9.status 2024-04-24T22:48:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:48:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_9.status 2024-04-24T22:48:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_9.status 2024-04-24T22:48:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_9.status 2024-04-24T22:48:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:48:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_9.status 2024-04-24T22:48:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2024-04-24T22:48:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T22:48:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2024-04-24T22:48:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 135 2024-04-24T22:48:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_9.status 2024-04-24T22:48:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:48:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T22:48:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_9.status 2024-04-24T22:48:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2024-04-24T22:48:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:57.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:48:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=135 2024-04-24T22:48:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:48:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_9.status 2024-04-24T22:48:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T22:48:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_9.status 2024-04-24T22:48:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-04-24T22:48:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:57.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:48:57.536 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-24T22:48:57.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-24T22:48:57.536 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-24T22:48:57.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-24T22:48:57.538 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2024-04-24T22:48:57.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:48:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:48:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:48:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:48:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 135 = '' 2024-04-24T22:48:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:48:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_9.status 2024-04-24T22:48:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T22:48:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_9.status 2024-04-24T22:48:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-04-24T22:48:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:48:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:48:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:48:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:48:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:48:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 135 = '' 2024-04-24T22:48:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:48:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:48:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_9.status 2024-04-24T22:48:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T22:48:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_9.status 2024-04-24T22:48:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-04-24T22:48:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:48:58.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:48:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=135 2024-04-24T22:48:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 135 = 135 2024-04-24T22:48:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-04-24T22:48:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:48:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T22:48:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:48:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:48:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:48:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-04-24T22:48:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_9': 2024-04-24T22:48:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:48:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:48:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:48:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc709d7e8a 2024-04-24T22:48:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc709d7e8a 2024-04-24T22:48:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:48:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:48:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:48:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:48:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:40:40 2024 2024-04-24T22:48:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:40:40 2024 2024-04-24T22:48:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:40:40 2024 2024-04-24T22:48:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:48:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:48:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:48:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 59c6ba05-b01f-4e71-ad9c-960605496d0d 2024-04-24T22:48:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:48:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:48:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_9 snap 2024-04-24T22:48:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:48:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T22:48:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:48:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:48:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_9.export 2024-04-24T22:48:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_9.export 2024-04-24T22:48:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_9.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_9.export 2024-04-24T22:48:59.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_9.export 2024-04-24T22:49:00.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:49:00.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_9.export 2024-04-24T22:49:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:49:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_9.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_9.export 2024-04-24T22:49:00.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_9.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_9.export 2024-04-24T22:49:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:49:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_10 2024-04-24T22:49:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_10 snap 2024-04-24T22:49:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T22:49:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:49:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-04-24T22:49:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T22:49:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-04-24T22:49:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T22:49:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:49:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:49:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:49:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:49:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-04-24T22:49:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T22:49:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:49:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:49:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:49:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-04-24T22:49:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:49:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:49:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:49:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:49:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:49:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:49:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:49:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:49:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:49:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-04-24T22:49:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:49:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:49:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:49:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:49:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:49:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:49:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:49:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:49:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:49:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:49:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:49:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:49:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:49:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:49:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:49:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:49:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:03.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-04-24T22:49:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T22:49:03.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:03.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2024-04-24T22:49:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:49:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:49:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:49:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:49:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:49:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:49:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:49:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:49:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:49:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:49:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:49:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:49:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:49:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:49:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:49:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:49:03.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:03.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:03.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:03.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2024-04-24T22:49:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T22:49:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2024-04-24T22:49:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:49:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:49:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:49:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:49:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:49:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:49:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:49:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:49:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:49:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:49:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:49:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:49:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:49:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:49:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:49:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:49:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2024-04-24T22:49:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-04-24T22:49:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_10 2024-04-24T22:49:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:49:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:49:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:49:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:49:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:49:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:49:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:49:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:49:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:49:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:49:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:49:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:49:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:49:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:49:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:49:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:49:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:03.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:03.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:03.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_10 2024-04-24T22:49:03.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:49:03.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:49:03.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:49:03.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:49:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:49:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:49:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:49:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2024-04-24T22:49:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:49:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T22:49:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:49:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:49:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2024-04-24T22:49:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:49:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T22:49:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_10.status 2024-04-24T22:49:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_10.status 2024-04-24T22:49:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:49:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_10.status 2024-04-24T22:49:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_10.status 2024-04-24T22:49:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_10.status 2024-04-24T22:49:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:49:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_10.status 2024-04-24T22:49:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2024-04-24T22:49:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T22:49:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2024-04-24T22:49:04.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 137 2024-04-24T22:49:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_10.status 2024-04-24T22:49:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T22:49:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_10.status 2024-04-24T22:49:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2024-04-24T22:49:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:49:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=137 2024-04-24T22:49:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:49:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:49:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_10.status 2024-04-24T22:49:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T22:49:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_10.status 2024-04-24T22:49:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-04-24T22:49:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:49:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:49:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:49:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:49:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:49:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 137 = '' 2024-04-24T22:49:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:49:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_10.status 2024-04-24T22:49:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T22:49:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_10.status 2024-04-24T22:49:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-04-24T22:49:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:05.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:49:05.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:49:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:49:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:49:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:49:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 137 = '' 2024-04-24T22:49:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:49:05.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_10.status 2024-04-24T22:49:05.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:05.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:05.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T22:49:05.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_10.status 2024-04-24T22:49:05.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-04-24T22:49:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=137 2024-04-24T22:49:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 137 = 137 2024-04-24T22:49:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-04-24T22:49:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T22:49:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:49:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:49:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:49:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-04-24T22:49:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_10': 2024-04-24T22:49:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:49:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:49:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:49:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc5b23b777 2024-04-24T22:49:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc5b23b777 2024-04-24T22:49:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:49:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:49:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:49:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:49:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:40:48 2024 2024-04-24T22:49:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:40:48 2024 2024-04-24T22:49:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:40:48 2024 2024-04-24T22:49:06.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:49:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:49:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:49:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 020b3f22-a97b-4ec7-aca6-b7942750a5f1 2024-04-24T22:49:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:49:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_10 snap 2024-04-24T22:49:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T22:49:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:49:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_10.export 2024-04-24T22:49:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_10.export 2024-04-24T22:49:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_10.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_10.export 2024-04-24T22:49:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_10.export 2024-04-24T22:49:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:49:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_10.export 2024-04-24T22:49:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:49:07.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_10.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_10.export 2024-04-24T22:49:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_10.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_10.export 2024-04-24T22:49:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:49:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_11 2024-04-24T22:49:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_11 snap 2024-04-24T22:49:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T22:49:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:49:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-04-24T22:49:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T22:49:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-04-24T22:49:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T22:49:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:49:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:49:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:49:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:49:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-04-24T22:49:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T22:49:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:49:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:49:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:49:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-04-24T22:49:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:49:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:49:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:49:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:49:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:49:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:49:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:49:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:49:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:49:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-04-24T22:49:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:49:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:49:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:49:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:49:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:49:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:49:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:49:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:49:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:49:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:49:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:49:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:49:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:49:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:49:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:49:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:49:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-04-24T22:49:10.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-04-24T22:49:10.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:10.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:10.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:10.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2024-04-24T22:49:10.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:49:10.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:10.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:49:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:49:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:49:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:49:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:49:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:49:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:49:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:49:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:49:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:49:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:49:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:49:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:49:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:49:10.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:49:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:10.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:10.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:10.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2024-04-24T22:49:10.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-04-24T22:49:10.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:10.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_11 2024-04-24T22:49:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:49:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:49:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:49:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:49:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:49:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:49:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:49:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:49:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:49:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:49:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:49:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:49:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:49:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:49:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:49:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:49:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_11 2024-04-24T22:49:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-04-24T22:49:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_11 2024-04-24T22:49:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:49:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:49:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:49:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:49:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:49:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:49:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:49:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:49:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:49:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:49:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:49:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:49:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:49:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:49:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:49:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:49:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_11 2024-04-24T22:49:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:49:10.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:49:10.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:49:10.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:49:10.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:49:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:49:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:49:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2024-04-24T22:49:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:49:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T22:49:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:49:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:49:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2024-04-24T22:49:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:49:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T22:49:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_11.status 2024-04-24T22:49:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_11.status 2024-04-24T22:49:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:49:10.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_11.status 2024-04-24T22:49:10.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_11.status 2024-04-24T22:49:10.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_11.status 2024-04-24T22:49:10.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:49:10.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_11.status 2024-04-24T22:49:10.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2024-04-24T22:49:10.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:10.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:10.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T22:49:10.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2024-04-24T22:49:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 139 2024-04-24T22:49:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_11.status 2024-04-24T22:49:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:11.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T22:49:11.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_11.status 2024-04-24T22:49:11.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2024-04-24T22:49:11.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:49:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=139 2024-04-24T22:49:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:49:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:49:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_11.status 2024-04-24T22:49:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T22:49:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_11.status 2024-04-24T22:49:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-04-24T22:49:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:49:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:49:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:49:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:49:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:49:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 139 = '' 2024-04-24T22:49:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:49:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_11.status 2024-04-24T22:49:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T22:49:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_11.status 2024-04-24T22:49:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-04-24T22:49:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:49:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:49:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:49:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:49:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:49:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 139 = '' 2024-04-24T22:49:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:49:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_11.status 2024-04-24T22:49:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T22:49:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_11.status 2024-04-24T22:49:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-04-24T22:49:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=139 2024-04-24T22:49:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 139 = 139 2024-04-24T22:49:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-04-24T22:49:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T22:49:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:49:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:49:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:49:13.659 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-24T22:49:13.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-24T22:49:13.660 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-04-24T22:49:13.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T22:49:13.659+0000 7fc150031700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 60192) UID: 1000 2024-04-24T22:49:13.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:49:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-04-24T22:49:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_11': 2024-04-24T22:49:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:49:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:49:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:49:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dcfcb8faf3 2024-04-24T22:49:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dcfcb8faf3 2024-04-24T22:49:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:49:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:49:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:49:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:49:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:40:59 2024 2024-04-24T22:49:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:40:59 2024 2024-04-24T22:49:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:40:59 2024 2024-04-24T22:49:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:49:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:49:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:49:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 8e1cc3c0-2ace-44c5-a115-11e73865c94e 2024-04-24T22:49:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:49:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_11 snap 2024-04-24T22:49:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T22:49:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:49:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_11.export 2024-04-24T22:49:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_11.export 2024-04-24T22:49:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_11.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_11.export 2024-04-24T22:49:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_11.export 2024-04-24T22:49:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:49:14.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_11.export 2024-04-24T22:49:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:49:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_11.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_11.export 2024-04-24T22:49:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_11.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_11.export 2024-04-24T22:49:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:49:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_12 2024-04-24T22:49:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_12 snap 2024-04-24T22:49:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T22:49:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:49:14.661 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-24T22:49:14.661 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:49:14.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:49:14.661 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-24T22:49:14.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-24T22:49:14.661 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-24T22:49:14.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-24T22:49:14.664 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2024-04-24T22:49:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-04-24T22:49:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T22:49:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-04-24T22:49:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T22:49:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:49:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:49:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:49:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:49:17.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-04-24T22:49:17.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:17.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:17.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T22:49:17.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:49:17.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:49:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:49:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-04-24T22:49:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:49:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:49:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:49:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:49:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:49:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:49:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:49:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:49:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:49:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-04-24T22:49:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:49:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:49:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:49:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:49:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:49:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:49:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:49:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:49:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:49:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:49:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:49:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:49:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:49:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:49:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:49:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:49:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-04-24T22:49:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-24T22:49:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2024-04-24T22:49:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:49:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:49:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:49:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:49:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:49:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:49:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:49:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:49:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:49:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:49:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:49:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:49:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:49:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:49:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:49:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:49:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2024-04-24T22:49:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-24T22:49:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2024-04-24T22:49:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:49:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:49:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:49:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:49:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:49:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:49:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:49:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:49:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:49:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:49:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:49:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:49:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:49:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:49:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:49:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:49:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2024-04-24T22:49:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-04-24T22:49:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_12 2024-04-24T22:49:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:49:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:49:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:49:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:49:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:49:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:49:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:49:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:49:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:49:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:49:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:49:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:49:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:49:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:49:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:49:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:49:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_12 2024-04-24T22:49:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:49:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:49:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:49:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:49:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:49:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:49:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:49:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2024-04-24T22:49:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:49:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T22:49:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:49:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:49:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2024-04-24T22:49:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:49:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T22:49:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_12.status 2024-04-24T22:49:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_12.status 2024-04-24T22:49:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:49:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_12.status 2024-04-24T22:49:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_12.status 2024-04-24T22:49:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_12.status 2024-04-24T22:49:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:49:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_12.status 2024-04-24T22:49:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2024-04-24T22:49:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T22:49:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2024-04-24T22:49:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 141 2024-04-24T22:49:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_12.status 2024-04-24T22:49:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T22:49:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_12.status 2024-04-24T22:49:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2024-04-24T22:49:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:49:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=141 2024-04-24T22:49:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:49:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:49:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_12.status 2024-04-24T22:49:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T22:49:18.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_12.status 2024-04-24T22:49:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-04-24T22:49:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:49:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:49:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:49:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:49:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:49:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 141 = '' 2024-04-24T22:49:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:49:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_12.status 2024-04-24T22:49:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T22:49:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_12.status 2024-04-24T22:49:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-04-24T22:49:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:49:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:49:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:49:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:49:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:49:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 141 = '' 2024-04-24T22:49:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:49:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_12.status 2024-04-24T22:49:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T22:49:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_12.status 2024-04-24T22:49:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-04-24T22:49:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=141 2024-04-24T22:49:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 141 = 141 2024-04-24T22:49:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-04-24T22:49:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T22:49:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:49:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:49:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:49:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-04-24T22:49:20.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_12': 2024-04-24T22:49:20.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:49:20.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:49:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:49:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dcbca86992 2024-04-24T22:49:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dcbca86992 2024-04-24T22:49:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:49:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:49:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:49:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:49:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:41:05 2024 2024-04-24T22:49:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:41:05 2024 2024-04-24T22:49:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:41:05 2024 2024-04-24T22:49:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:49:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:49:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:49:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 95e4bc86-7500-4354-96f7-2ccfc5fb3d66 2024-04-24T22:49:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:49:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_12 snap 2024-04-24T22:49:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T22:49:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:49:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_12.export 2024-04-24T22:49:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_12.export 2024-04-24T22:49:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_12.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_12.export 2024-04-24T22:49:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_12.export 2024-04-24T22:49:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:49:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_12.export 2024-04-24T22:49:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:49:21.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_12.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_12.export 2024-04-24T22:49:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_12.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_12.export 2024-04-24T22:49:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:49:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_13 2024-04-24T22:49:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_13 snap 2024-04-24T22:49:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T22:49:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:49:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-04-24T22:49:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T22:49:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-04-24T22:49:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T22:49:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:49:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:49:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:49:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:49:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-04-24T22:49:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T22:49:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:49:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:49:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:49:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-04-24T22:49:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:49:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:49:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:49:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:49:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:49:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:49:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:49:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:49:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:49:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-04-24T22:49:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:49:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:49:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:49:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:49:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:49:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:49:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:49:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:49:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:49:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:49:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:49:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:49:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:49:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:49:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:49:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:49:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-04-24T22:49:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-04-24T22:49:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2024-04-24T22:49:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:49:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:49:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:49:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:49:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:49:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:49:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:49:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:49:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:49:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:49:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:49:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:49:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:49:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:49:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:49:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:49:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2024-04-24T22:49:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-04-24T22:49:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_13 2024-04-24T22:49:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:49:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:49:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:49:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:49:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:49:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:49:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:49:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:49:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:49:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:49:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:49:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:49:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:49:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:49:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:49:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:49:24.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:24.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:24.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:24.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_13 2024-04-24T22:49:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-04-24T22:49:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_13 2024-04-24T22:49:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:49:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:49:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:49:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:49:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:49:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:49:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:49:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:49:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:49:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:49:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:49:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:49:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:49:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:49:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:49:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:49:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_13 2024-04-24T22:49:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:49:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:49:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:49:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:49:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:49:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:49:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:49:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2024-04-24T22:49:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:49:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T22:49:24.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:49:24.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:49:24.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2024-04-24T22:49:24.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:49:24.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:24.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:24.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T22:49:24.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_13.status 2024-04-24T22:49:24.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_13.status 2024-04-24T22:49:24.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:49:24.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_13.status 2024-04-24T22:49:24.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_13.status 2024-04-24T22:49:24.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_13.status 2024-04-24T22:49:24.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:49:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_13.status 2024-04-24T22:49:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2024-04-24T22:49:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T22:49:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2024-04-24T22:49:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 143 2024-04-24T22:49:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_13.status 2024-04-24T22:49:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T22:49:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_13.status 2024-04-24T22:49:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2024-04-24T22:49:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:49:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=143 2024-04-24T22:49:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:49:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:49:25.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_13.status 2024-04-24T22:49:25.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:25.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:25.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T22:49:25.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_13.status 2024-04-24T22:49:25.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-04-24T22:49:25.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:49:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:49:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:49:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:49:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:49:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 143 = '' 2024-04-24T22:49:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:49:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_13.status 2024-04-24T22:49:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T22:49:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_13.status 2024-04-24T22:49:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-04-24T22:49:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:26.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:49:26.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:49:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:49:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:49:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:49:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 143 = '' 2024-04-24T22:49:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:49:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_13.status 2024-04-24T22:49:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T22:49:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_13.status 2024-04-24T22:49:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-04-24T22:49:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=143 2024-04-24T22:49:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 143 = 143 2024-04-24T22:49:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-04-24T22:49:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T22:49:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:49:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:49:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:49:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-04-24T22:49:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_13': 2024-04-24T22:49:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:49:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:49:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:49:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc8e9b9736 2024-04-24T22:49:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc8e9b9736 2024-04-24T22:49:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:49:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:49:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:49:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:49:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:41:17 2024 2024-04-24T22:49:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:41:17 2024 2024-04-24T22:49:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:41:17 2024 2024-04-24T22:49:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:49:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:49:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:49:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 4bd2c65f-f68d-4a16-b6d4-a7ab70837e25 2024-04-24T22:49:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:49:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_13 snap 2024-04-24T22:49:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T22:49:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:49:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_13.export 2024-04-24T22:49:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_13.export 2024-04-24T22:49:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_13.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_13.export 2024-04-24T22:49:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_13.export 2024-04-24T22:49:28.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:49:28.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_13.export 2024-04-24T22:49:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:49:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_13.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_13.export 2024-04-24T22:49:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_13.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_13.export 2024-04-24T22:49:28.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:28.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:49:28.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_14 2024-04-24T22:49:28.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_14 snap 2024-04-24T22:49:28.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:28.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T22:49:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:49:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-04-24T22:49:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T22:49:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-04-24T22:49:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T22:49:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:49:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:49:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:49:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:49:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-04-24T22:49:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T22:49:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:49:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:49:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:49:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-04-24T22:49:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:49:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:49:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:49:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:49:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:49:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:49:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:49:30.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:49:30.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:49:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-04-24T22:49:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:49:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:49:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:49:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:49:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:49:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:49:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:49:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:49:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:49:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:49:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:49:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:49:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:49:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:49:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:49:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:49:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-04-24T22:49:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-24T22:49:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2024-04-24T22:49:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:49:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:49:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:49:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:49:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:49:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:49:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:49:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:49:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:49:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:49:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:49:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:49:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:49:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:49:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:49:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:49:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2024-04-24T22:49:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-24T22:49:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_14 2024-04-24T22:49:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:49:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:49:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:49:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:49:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:49:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:49:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:49:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:49:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:49:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:49:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:49:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:49:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:49:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:49:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:49:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:49:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_14 2024-04-24T22:49:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-04-24T22:49:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_14 2024-04-24T22:49:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:49:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:49:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:49:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:49:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:49:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:49:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:49:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:49:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:49:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:49:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:49:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:49:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:49:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:49:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:49:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:49:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_14 2024-04-24T22:49:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:49:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:49:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:49:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:49:31.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:49:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:49:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:49:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2024-04-24T22:49:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:49:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T22:49:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:49:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:49:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2024-04-24T22:49:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:49:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T22:49:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_14.status 2024-04-24T22:49:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_14.status 2024-04-24T22:49:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:49:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_14.status 2024-04-24T22:49:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_14.status 2024-04-24T22:49:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_14.status 2024-04-24T22:49:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:49:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_14.status 2024-04-24T22:49:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2024-04-24T22:49:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T22:49:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2024-04-24T22:49:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 145 2024-04-24T22:49:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_14.status 2024-04-24T22:49:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T22:49:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_14.status 2024-04-24T22:49:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2024-04-24T22:49:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:49:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=145 2024-04-24T22:49:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:49:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:49:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_14.status 2024-04-24T22:49:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T22:49:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_14.status 2024-04-24T22:49:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-04-24T22:49:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=81 2024-04-24T22:49:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 145 = 81 2024-04-24T22:49:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:49:32.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_14.status 2024-04-24T22:49:32.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:32.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:32.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T22:49:32.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_14.status 2024-04-24T22:49:32.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-04-24T22:49:32.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=81 2024-04-24T22:49:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 145 = 81 2024-04-24T22:49:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:49:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_14.status 2024-04-24T22:49:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T22:49:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_14.status 2024-04-24T22:49:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-04-24T22:49:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:49:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:33.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:49:33.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:49:33.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:49:33.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:49:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 145 = '' 2024-04-24T22:49:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T22:49:35.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_14.status 2024-04-24T22:49:35.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T22:49:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_14.status 2024-04-24T22:49:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-04-24T22:49:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=145 2024-04-24T22:49:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 145 = 145 2024-04-24T22:49:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-04-24T22:49:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T22:49:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:49:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:49:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:49:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-04-24T22:49:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_14': 2024-04-24T22:49:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:49:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:49:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:49:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc5151f4dd 2024-04-24T22:49:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc5151f4dd 2024-04-24T22:49:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:49:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:49:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:49:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:49:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:41:30 2024 2024-04-24T22:49:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:41:30 2024 2024-04-24T22:49:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:41:30 2024 2024-04-24T22:49:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:49:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:49:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:49:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: e99fde4e-a334-4491-9ca4-d28b703e6700 2024-04-24T22:49:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:49:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_14 snap 2024-04-24T22:49:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T22:49:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:49:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_14.export 2024-04-24T22:49:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_14.export 2024-04-24T22:49:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_14.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_14.export 2024-04-24T22:49:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_14.export 2024-04-24T22:49:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:49:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_14.export 2024-04-24T22:49:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:49:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_14.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_14.export 2024-04-24T22:49:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_14.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_14.export 2024-04-24T22:49:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:49:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_15 2024-04-24T22:49:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_15 snap 2024-04-24T22:49:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T22:49:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:49:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-04-24T22:49:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T22:49:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-04-24T22:49:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T22:49:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:49:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:49:38.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:49:38.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:49:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-04-24T22:49:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T22:49:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:49:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:49:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:49:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-04-24T22:49:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:49:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:49:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:49:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:49:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:49:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:49:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:49:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:49:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:49:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-04-24T22:49:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:49:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:49:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:49:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:49:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:49:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:49:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:49:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:49:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:49:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:49:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:49:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:49:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:49:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:49:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:49:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:49:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-04-24T22:49:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-24T22:49:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:39.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2024-04-24T22:49:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:49:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:49:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:49:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:49:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:49:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:49:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:49:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:49:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:49:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:49:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:49:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:49:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:49:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:49:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:49:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:49:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2024-04-24T22:49:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-24T22:49:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_15 2024-04-24T22:49:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:49:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:49:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:49:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:49:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:49:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:49:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:49:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:49:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:49:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:49:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:49:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:49:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:49:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:49:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:49:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:49:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_15 2024-04-24T22:49:39.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-04-24T22:49:39.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:39.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_15 2024-04-24T22:49:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:49:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:49:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:49:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:49:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:49:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:49:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:49:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:49:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:49:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:49:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:49:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:49:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:49:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:49:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:49:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:49:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_15 2024-04-24T22:49:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:49:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:49:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:49:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:49:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:49:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:49:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:49:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2024-04-24T22:49:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:49:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T22:49:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:49:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:49:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2024-04-24T22:49:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:49:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T22:49:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_15.status 2024-04-24T22:49:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_15.status 2024-04-24T22:49:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:49:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_15.status 2024-04-24T22:49:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_15.status 2024-04-24T22:49:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_15.status 2024-04-24T22:49:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:49:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_15.status 2024-04-24T22:49:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2024-04-24T22:49:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T22:49:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2024-04-24T22:49:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 147 2024-04-24T22:49:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_15.status 2024-04-24T22:49:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T22:49:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_15.status 2024-04-24T22:49:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2024-04-24T22:49:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:49:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=147 2024-04-24T22:49:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:49:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:40.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:49:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_15.status 2024-04-24T22:49:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T22:49:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_15.status 2024-04-24T22:49:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-04-24T22:49:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=82 2024-04-24T22:49:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 147 = 82 2024-04-24T22:49:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:40.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:49:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_15.status 2024-04-24T22:49:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T22:49:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_15.status 2024-04-24T22:49:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-04-24T22:49:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:49:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:49:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:49:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:49:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:49:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 147 = '' 2024-04-24T22:49:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:49:41.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_15.status 2024-04-24T22:49:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T22:49:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_15.status 2024-04-24T22:49:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-04-24T22:49:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=147 2024-04-24T22:49:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 147 = 147 2024-04-24T22:49:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-04-24T22:49:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T22:49:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:49:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:49:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:49:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-04-24T22:49:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_15': 2024-04-24T22:49:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:49:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:49:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:49:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc9ade28ce 2024-04-24T22:49:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc9ade28ce 2024-04-24T22:49:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:49:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:49:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:49:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:49:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:41:37 2024 2024-04-24T22:49:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:41:37 2024 2024-04-24T22:49:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:41:37 2024 2024-04-24T22:49:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:49:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:49:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:49:42.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 61a0be27-386b-497f-bf17-b0030cfd9593 2024-04-24T22:49:42.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:49:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_15 snap 2024-04-24T22:49:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T22:49:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:49:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_15.export 2024-04-24T22:49:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_15.export 2024-04-24T22:49:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_15.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_15.export 2024-04-24T22:49:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_15.export 2024-04-24T22:49:43.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:49:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_15.export 2024-04-24T22:49:43.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:49:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_15.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_15.export 2024-04-24T22:49:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_15.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_15.export 2024-04-24T22:49:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:49:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_16 2024-04-24T22:49:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_16 snap 2024-04-24T22:49:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T22:49:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:49:45.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-04-24T22:49:45.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T22:49:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-04-24T22:49:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T22:49:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:49:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:49:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:49:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:49:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-04-24T22:49:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T22:49:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:49:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:49:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:49:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-04-24T22:49:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:49:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:49:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:49:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:49:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:49:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:49:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:49:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:49:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:49:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-04-24T22:49:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:49:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:49:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:49:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:49:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:49:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:49:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:49:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:49:46.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:49:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:49:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:49:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:49:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:49:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:49:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:49:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:49:46.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-04-24T22:49:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-24T22:49:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-04-24T22:49:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:49:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:49:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:49:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:49:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:49:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:49:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:49:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:49:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:49:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:49:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:49:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:49:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:49:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:49:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:49:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:49:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-04-24T22:49:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-24T22:49:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2024-04-24T22:49:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:49:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:49:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:49:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:49:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:49:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:49:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:49:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:49:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:49:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:49:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:49:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:49:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:49:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:49:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:49:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:49:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2024-04-24T22:49:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-04-24T22:49:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_16 2024-04-24T22:49:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:49:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:49:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:49:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:49:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:49:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:49:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:49:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:49:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:49:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:49:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:49:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:49:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:49:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:49:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:49:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:49:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_16 2024-04-24T22:49:46.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:49:46.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:49:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:49:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:49:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:49:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:49:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:49:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2024-04-24T22:49:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:49:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T22:49:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:49:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:49:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2024-04-24T22:49:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:49:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:46.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T22:49:46.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_16.status 2024-04-24T22:49:46.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_16.status 2024-04-24T22:49:46.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:49:46.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_16.status 2024-04-24T22:49:46.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_16.status 2024-04-24T22:49:46.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_16.status 2024-04-24T22:49:46.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:49:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_16.status 2024-04-24T22:49:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2024-04-24T22:49:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T22:49:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2024-04-24T22:49:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 149 2024-04-24T22:49:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_16.status 2024-04-24T22:49:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T22:49:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_16.status 2024-04-24T22:49:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2024-04-24T22:49:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:49:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=149 2024-04-24T22:49:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:49:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:49:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_16.status 2024-04-24T22:49:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T22:49:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_16.status 2024-04-24T22:49:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-04-24T22:49:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=83 2024-04-24T22:49:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 149 = 83 2024-04-24T22:49:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:49:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_16.status 2024-04-24T22:49:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T22:49:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_16.status 2024-04-24T22:49:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-04-24T22:49:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:48.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:49:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:49:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:49:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:49:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:49:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 149 = '' 2024-04-24T22:49:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:49:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_16.status 2024-04-24T22:49:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T22:49:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_16.status 2024-04-24T22:49:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-04-24T22:49:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=149 2024-04-24T22:49:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 149 = 149 2024-04-24T22:49:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-04-24T22:49:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T22:49:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:49:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:49:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:49:49.629 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:49:49.627+0000 7fd668aa0700 -1 reset not still connected to 0x56391f5e8a90 2024-04-24T22:49:49.630 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:49:49.627+0000 7fd668aa0700 -1 reset not still connected to 0x56391f85b6c0 2024-04-24T22:49:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-04-24T22:49:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_16': 2024-04-24T22:49:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:49:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:49:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:49:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dcd9ff99ef 2024-04-24T22:49:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dcd9ff99ef 2024-04-24T22:49:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:49:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:49:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:49:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:49:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:41:48 2024 2024-04-24T22:49:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:41:48 2024 2024-04-24T22:49:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:41:48 2024 2024-04-24T22:49:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:49:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:49:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:49:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 73832a9a-0532-4696-9e85-d0aaea380e6c 2024-04-24T22:49:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:49:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_16 snap 2024-04-24T22:49:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T22:49:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:49:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_16.export 2024-04-24T22:49:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_16.export 2024-04-24T22:49:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_16.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_16.export 2024-04-24T22:49:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_16.export 2024-04-24T22:49:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:49:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_16.export 2024-04-24T22:49:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:49:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_16.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_16.export 2024-04-24T22:49:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_16.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_16.export 2024-04-24T22:49:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:49:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_17 2024-04-24T22:49:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_17 snap 2024-04-24T22:49:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T22:49:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:49:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-04-24T22:49:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T22:49:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-04-24T22:49:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T22:49:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:49:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:49:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:49:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:49:53.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-04-24T22:49:53.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:53.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:53.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T22:49:53.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:49:53.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:49:53.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:49:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-04-24T22:49:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:49:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:49:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:49:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:49:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:49:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:49:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:49:53.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:49:53.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:49:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-04-24T22:49:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:49:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:49:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:49:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:49:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:49:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:49:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:49:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:49:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:49:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:49:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:49:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:49:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:49:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:49:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:49:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:49:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:49:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-04-24T22:49:53.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-04-24T22:49:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2024-04-24T22:49:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:49:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:49:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:49:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:49:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:49:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:49:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:49:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:49:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:49:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:49:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:49:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:53.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:49:53.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:49:53.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:53.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:53.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:49:53.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:49:53.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:49:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:53.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:49:53.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2024-04-24T22:49:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-04-24T22:49:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_17 2024-04-24T22:49:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:49:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:49:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:49:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:49:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:49:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:49:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:49:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:49:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:49:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:49:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:49:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:49:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:49:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:49:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:49:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:49:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:49:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_17 2024-04-24T22:49:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-04-24T22:49:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:49:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:49:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:49:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_17 2024-04-24T22:49:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:49:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:49:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:49:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:49:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:49:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:49:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:49:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:49:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:49:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:49:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:49:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:49:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:49:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:49:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:49:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:49:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:49:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:49:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:49:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:49:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:49:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:49:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:49:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:49:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:49:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:49:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:49:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:49:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:49:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:49:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_17 2024-04-24T22:49:53.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:49:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:49:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:49:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:49:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:49:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:49:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:49:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2024-04-24T22:49:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:49:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T22:49:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:49:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:49:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2024-04-24T22:49:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:49:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T22:49:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_17.status 2024-04-24T22:49:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_17.status 2024-04-24T22:49:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:49:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_17.status 2024-04-24T22:49:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_17.status 2024-04-24T22:49:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_17.status 2024-04-24T22:49:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:49:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_17.status 2024-04-24T22:49:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2024-04-24T22:49:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T22:49:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2024-04-24T22:49:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 151 2024-04-24T22:49:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_17.status 2024-04-24T22:49:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T22:49:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_17.status 2024-04-24T22:49:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2024-04-24T22:49:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:49:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=151 2024-04-24T22:49:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:49:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:49:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_17.status 2024-04-24T22:49:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T22:49:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_17.status 2024-04-24T22:49:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-04-24T22:49:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=84 2024-04-24T22:49:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 151 = 84 2024-04-24T22:49:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:49:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_17.status 2024-04-24T22:49:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T22:49:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_17.status 2024-04-24T22:49:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-04-24T22:49:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:49:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:55.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:49:55.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:49:55.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:49:55.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:49:55.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 151 = '' 2024-04-24T22:49:55.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:49:55.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:49:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_17.status 2024-04-24T22:49:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T22:49:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_17.status 2024-04-24T22:49:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-04-24T22:49:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:49:55.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:49:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=151 2024-04-24T22:49:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 151 = 151 2024-04-24T22:49:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-04-24T22:49:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T22:49:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:49:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:49:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:49:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-04-24T22:49:56.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_17': 2024-04-24T22:49:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:49:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:49:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:49:56.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dcd0540a0b 2024-04-24T22:49:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dcd0540a0b 2024-04-24T22:49:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:49:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:49:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:49:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:49:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:41:54 2024 2024-04-24T22:49:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:41:54 2024 2024-04-24T22:49:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:41:54 2024 2024-04-24T22:49:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:49:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:49:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:49:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: ea68e336-4843-42cb-80ec-9f3fa4c17a25 2024-04-24T22:49:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:49:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_17 snap 2024-04-24T22:49:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T22:49:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:49:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_17.export 2024-04-24T22:49:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_17.export 2024-04-24T22:49:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_17.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_17.export 2024-04-24T22:49:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_17.export 2024-04-24T22:49:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:49:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_17.export 2024-04-24T22:49:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:49:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_17.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_17.export 2024-04-24T22:49:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_17.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_17.export 2024-04-24T22:49:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:49:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:49:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_18 2024-04-24T22:49:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_18 snap 2024-04-24T22:49:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:49:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T22:49:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:49:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:49:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-04-24T22:49:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T22:49:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-04-24T22:49:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:49:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:49:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T22:49:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:49:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:49:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:49:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:50:00.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-04-24T22:50:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T22:50:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:50:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:50:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:50:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-04-24T22:50:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:50:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:50:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:50:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:50:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:50:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:50:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:50:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:50:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:50:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-04-24T22:50:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:50:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:50:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:50:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:50:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:50:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:50:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:50:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:50:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:50:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:50:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:50:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:50:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:50:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:50:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:50:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:50:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-04-24T22:50:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-04-24T22:50:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2024-04-24T22:50:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:50:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:50:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:50:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:50:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:50:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:50:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:50:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:50:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:50:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:50:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:50:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:50:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:50:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:00.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:50:00.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:50:00.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:50:00.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:00.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:00.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:00.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2024-04-24T22:50:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-04-24T22:50:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_18 2024-04-24T22:50:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:50:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:50:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:50:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:50:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:50:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:50:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:50:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:50:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:50:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:50:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:50:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:50:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:50:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:50:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:50:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:50:00.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:00.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:00.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:00.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_18 2024-04-24T22:50:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-04-24T22:50:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_18 2024-04-24T22:50:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:50:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:50:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:50:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:50:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:50:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:50:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:50:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:50:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:50:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:50:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:50:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:50:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:50:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:50:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:50:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:50:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_18 2024-04-24T22:50:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:50:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:50:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:50:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:50:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:50:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:50:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:50:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2024-04-24T22:50:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:50:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T22:50:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:50:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:50:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2024-04-24T22:50:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:50:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T22:50:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_18.status 2024-04-24T22:50:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_18.status 2024-04-24T22:50:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:50:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_18.status 2024-04-24T22:50:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_18.status 2024-04-24T22:50:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_18.status 2024-04-24T22:50:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:50:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_18.status 2024-04-24T22:50:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2024-04-24T22:50:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T22:50:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2024-04-24T22:50:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 153 2024-04-24T22:50:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_18.status 2024-04-24T22:50:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T22:50:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_18.status 2024-04-24T22:50:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2024-04-24T22:50:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:01.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:50:01.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=153 2024-04-24T22:50:01.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:01.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:01.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:50:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_18.status 2024-04-24T22:50:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T22:50:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_18.status 2024-04-24T22:50:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-04-24T22:50:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:50:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:50:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:50:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:50:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 153 = '' 2024-04-24T22:50:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:50:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_18.status 2024-04-24T22:50:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T22:50:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_18.status 2024-04-24T22:50:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-04-24T22:50:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:02.026 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:02.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:50:02.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:50:02.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:50:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:50:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 153 = '' 2024-04-24T22:50:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:50:02.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_18.status 2024-04-24T22:50:02.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:02.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:02.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T22:50:02.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_18.status 2024-04-24T22:50:02.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-04-24T22:50:02.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=153 2024-04-24T22:50:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 153 = 153 2024-04-24T22:50:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-04-24T22:50:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T22:50:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:50:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:50:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:50:03.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-04-24T22:50:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_18': 2024-04-24T22:50:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:50:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:50:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:50:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc5c64e94a 2024-04-24T22:50:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc5c64e94a 2024-04-24T22:50:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:50:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:50:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:50:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:50:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:42:08 2024 2024-04-24T22:50:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:42:08 2024 2024-04-24T22:50:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:42:08 2024 2024-04-24T22:50:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:50:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:50:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:50:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: e52f7b2d-1d65-436d-bbae-a4af8a5d1482 2024-04-24T22:50:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:50:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_18 snap 2024-04-24T22:50:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T22:50:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:50:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_18.export 2024-04-24T22:50:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_18.export 2024-04-24T22:50:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_18.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_18.export 2024-04-24T22:50:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_18.export 2024-04-24T22:50:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:50:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_18.export 2024-04-24T22:50:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:50:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_18.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_18.export 2024-04-24T22:50:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_18.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_18.export 2024-04-24T22:50:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:50:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_19 2024-04-24T22:50:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_19 snap 2024-04-24T22:50:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T22:50:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:50:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-04-24T22:50:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T22:50:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-04-24T22:50:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T22:50:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:50:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:50:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:50:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:50:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-04-24T22:50:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T22:50:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:50:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:50:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:50:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-04-24T22:50:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:50:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:50:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:50:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:50:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:50:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:50:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:50:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:50:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:50:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-04-24T22:50:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:50:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:50:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:50:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:50:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:50:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:50:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:50:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:50:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:50:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:50:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:50:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:50:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:50:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:50:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:50:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:50:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-04-24T22:50:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-04-24T22:50:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2024-04-24T22:50:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:50:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:50:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:50:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:50:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:50:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:50:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:50:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:50:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:50:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:50:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:50:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:50:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:50:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:50:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:50:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:50:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2024-04-24T22:50:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-04-24T22:50:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_19 2024-04-24T22:50:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:50:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:50:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:50:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:50:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:50:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:50:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:50:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:50:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:50:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:50:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:07.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_19 2024-04-24T22:50:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-04-24T22:50:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:07.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:07.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_19 2024-04-24T22:50:07.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:50:07.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:07.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:07.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:50:07.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:50:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:50:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:50:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:50:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:50:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:50:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:50:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:50:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:50:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:50:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:50:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:50:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:50:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:50:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_19 2024-04-24T22:50:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:50:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:50:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:50:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:50:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:50:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2024-04-24T22:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T22:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2024-04-24T22:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:50:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T22:50:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_19.status 2024-04-24T22:50:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_19.status 2024-04-24T22:50:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:50:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_19.status 2024-04-24T22:50:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_19.status 2024-04-24T22:50:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_19.status 2024-04-24T22:50:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:50:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_19.status 2024-04-24T22:50:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2024-04-24T22:50:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T22:50:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2024-04-24T22:50:08.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 155 2024-04-24T22:50:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_19.status 2024-04-24T22:50:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T22:50:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_19.status 2024-04-24T22:50:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2024-04-24T22:50:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:50:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=155 2024-04-24T22:50:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:50:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_19.status 2024-04-24T22:50:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T22:50:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_19.status 2024-04-24T22:50:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-04-24T22:50:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:08.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:50:08.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:50:08.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:50:08.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:50:08.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 155 = '' 2024-04-24T22:50:08.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:08.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:50:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_19.status 2024-04-24T22:50:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T22:50:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_19.status 2024-04-24T22:50:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-04-24T22:50:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:50:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:50:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:50:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:50:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 155 = '' 2024-04-24T22:50:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:50:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_19.status 2024-04-24T22:50:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T22:50:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_19.status 2024-04-24T22:50:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-04-24T22:50:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:09.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=155 2024-04-24T22:50:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 155 = 155 2024-04-24T22:50:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-04-24T22:50:09.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:09.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:09.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T22:50:09.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:09.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:50:09.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:50:09.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:50:10.354 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:50:10.352+0000 7fd668aa0700 -1 reset not still connected to 0x56391f5e8a90 2024-04-24T22:50:10.354 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:50:10.352+0000 7fd668aa0700 -1 reset not still connected to 0x56391f85b6c0 2024-04-24T22:50:10.355 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:50:10.352+0000 7fd668aa0700 -1 reset not still connected to 0x56392c547110 2024-04-24T22:50:10.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-04-24T22:50:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_19': 2024-04-24T22:50:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:50:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:50:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:50:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc725f2b4f 2024-04-24T22:50:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc725f2b4f 2024-04-24T22:50:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:50:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:50:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:50:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:50:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:42:15 2024 2024-04-24T22:50:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:42:15 2024 2024-04-24T22:50:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:42:15 2024 2024-04-24T22:50:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:50:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:50:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:50:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 38210526-ffd7-4449-be03-5e75b6c23176 2024-04-24T22:50:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:50:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_19 snap 2024-04-24T22:50:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T22:50:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:50:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_19.export 2024-04-24T22:50:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_19.export 2024-04-24T22:50:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_19.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_19.export 2024-04-24T22:50:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_19.export 2024-04-24T22:50:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:50:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_19.export 2024-04-24T22:50:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:50:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_19.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_19.export 2024-04-24T22:50:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_19.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_19.export 2024-04-24T22:50:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:50:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_20 2024-04-24T22:50:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_20 snap 2024-04-24T22:50:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T22:50:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:50:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-04-24T22:50:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T22:50:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-04-24T22:50:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T22:50:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:50:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:50:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:50:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:50:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-04-24T22:50:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T22:50:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:50:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:50:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:50:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-04-24T22:50:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:50:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:50:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:50:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:50:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:50:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:50:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:50:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:50:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:50:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-04-24T22:50:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:50:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:50:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:50:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:50:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:50:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:50:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:50:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:50:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:50:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:50:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:50:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:50:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:50:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:50:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:50:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:50:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-04-24T22:50:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-24T22:50:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-04-24T22:50:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:50:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:50:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:50:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:50:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:50:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:50:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:50:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:50:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:50:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:50:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:50:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:50:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:50:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:50:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:50:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:50:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-04-24T22:50:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-24T22:50:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:14.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2024-04-24T22:50:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:50:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:50:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:50:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:50:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:50:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:50:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:50:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:50:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:50:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:50:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:50:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:50:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:50:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:50:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:50:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:50:14.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2024-04-24T22:50:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-04-24T22:50:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_20 2024-04-24T22:50:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:50:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:50:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:50:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:50:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:50:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:50:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:50:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:50:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_20 2024-04-24T22:50:14.771 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:50:14.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:50:14.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:50:14.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:50:14.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:50:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:50:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:50:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2024-04-24T22:50:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:50:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T22:50:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:50:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:50:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2024-04-24T22:50:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:50:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T22:50:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_20.status 2024-04-24T22:50:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_20.status 2024-04-24T22:50:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:50:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_20.status 2024-04-24T22:50:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_20.status 2024-04-24T22:50:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_20.status 2024-04-24T22:50:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:50:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_20.status 2024-04-24T22:50:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2024-04-24T22:50:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T22:50:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2024-04-24T22:50:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 157 2024-04-24T22:50:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_20.status 2024-04-24T22:50:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T22:50:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_20.status 2024-04-24T22:50:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2024-04-24T22:50:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:50:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=157 2024-04-24T22:50:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:50:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_20.status 2024-04-24T22:50:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T22:50:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_20.status 2024-04-24T22:50:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-04-24T22:50:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:15.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:15.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:15.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:50:15.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:50:15.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:50:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:50:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 157 = '' 2024-04-24T22:50:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:50:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_20.status 2024-04-24T22:50:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T22:50:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_20.status 2024-04-24T22:50:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-04-24T22:50:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:50:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:50:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:50:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:50:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 157 = '' 2024-04-24T22:50:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:50:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_20.status 2024-04-24T22:50:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T22:50:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_20.status 2024-04-24T22:50:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-04-24T22:50:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=157 2024-04-24T22:50:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 157 = 157 2024-04-24T22:50:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-04-24T22:50:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T22:50:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:50:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:50:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:50:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-04-24T22:50:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_20': 2024-04-24T22:50:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:50:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:50:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:50:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc6a543956 2024-04-24T22:50:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc6a543956 2024-04-24T22:50:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:50:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:50:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:50:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:50:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:42:23 2024 2024-04-24T22:50:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:42:23 2024 2024-04-24T22:50:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:42:23 2024 2024-04-24T22:50:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:50:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:50:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:50:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 061306d6-3c6a-4f11-84ad-c1042b24a00e 2024-04-24T22:50:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:50:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_20 snap 2024-04-24T22:50:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T22:50:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:50:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_20.export 2024-04-24T22:50:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_20.export 2024-04-24T22:50:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_20.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_20.export 2024-04-24T22:50:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_20.export 2024-04-24T22:50:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:50:18.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_20.export 2024-04-24T22:50:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:50:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_20.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_20.export 2024-04-24T22:50:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_20.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_20.export 2024-04-24T22:50:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:50:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_21 2024-04-24T22:50:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_21 snap 2024-04-24T22:50:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T22:50:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:50:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-04-24T22:50:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T22:50:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-04-24T22:50:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T22:50:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:50:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:50:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:50:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:50:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-04-24T22:50:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T22:50:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:50:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:50:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:50:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-04-24T22:50:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:50:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:50:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:50:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:50:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:50:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:50:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:50:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:50:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:50:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-04-24T22:50:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:50:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:50:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:50:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:50:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:50:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:50:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:50:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:50:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:50:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:50:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:50:21.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:21.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:21.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:21.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:50:21.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:50:21.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:21.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:21.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:50:21.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:50:21.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:50:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-04-24T22:50:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-04-24T22:50:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2024-04-24T22:50:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:50:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:50:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:50:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:50:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:50:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:50:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:50:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:50:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:50:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:50:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:50:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:50:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:50:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:50:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:50:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:50:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2024-04-24T22:50:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-04-24T22:50:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_21 2024-04-24T22:50:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:50:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:50:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:50:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:50:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:50:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:50:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:50:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:50:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:50:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:50:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:50:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:50:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:50:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:50:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:50:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:50:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_21 2024-04-24T22:50:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-04-24T22:50:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:21.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_21 2024-04-24T22:50:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:50:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:50:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:50:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:50:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:50:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:50:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:50:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:50:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:50:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:50:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:50:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:50:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:50:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:50:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:50:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:50:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_21 2024-04-24T22:50:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:50:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:50:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:50:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:50:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:50:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:50:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:50:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2024-04-24T22:50:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:50:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T22:50:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:50:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:50:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2024-04-24T22:50:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:50:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T22:50:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_21.status 2024-04-24T22:50:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_21.status 2024-04-24T22:50:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:50:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_21.status 2024-04-24T22:50:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_21.status 2024-04-24T22:50:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_21.status 2024-04-24T22:50:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:50:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_21.status 2024-04-24T22:50:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2024-04-24T22:50:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T22:50:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2024-04-24T22:50:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 159 2024-04-24T22:50:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_21.status 2024-04-24T22:50:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T22:50:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_21.status 2024-04-24T22:50:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2024-04-24T22:50:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:50:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=159 2024-04-24T22:50:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:50:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_21.status 2024-04-24T22:50:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T22:50:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_21.status 2024-04-24T22:50:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-04-24T22:50:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:50:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:50:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:50:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:50:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 159 = '' 2024-04-24T22:50:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:50:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_21.status 2024-04-24T22:50:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T22:50:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_21.status 2024-04-24T22:50:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-04-24T22:50:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:23.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:23.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:23.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:50:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:50:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:50:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:50:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 159 = '' 2024-04-24T22:50:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:50:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_21.status 2024-04-24T22:50:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T22:50:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_21.status 2024-04-24T22:50:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-04-24T22:50:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=159 2024-04-24T22:50:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 159 = 159 2024-04-24T22:50:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-04-24T22:50:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T22:50:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:50:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:50:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:50:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-04-24T22:50:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_21': 2024-04-24T22:50:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:50:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:50:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:50:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dcf9fe12d9 2024-04-24T22:50:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dcf9fe12d9 2024-04-24T22:50:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:50:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:50:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:50:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:50:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:42:31 2024 2024-04-24T22:50:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:42:31 2024 2024-04-24T22:50:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:42:31 2024 2024-04-24T22:50:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:50:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:50:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:50:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 90338b6b-d62c-42ea-9b32-905a5d005824 2024-04-24T22:50:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:50:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_21 snap 2024-04-24T22:50:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T22:50:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:50:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_21.export 2024-04-24T22:50:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_21.export 2024-04-24T22:50:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_21.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_21.export 2024-04-24T22:50:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_21.export 2024-04-24T22:50:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:50:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_21.export 2024-04-24T22:50:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:50:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_21.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_21.export 2024-04-24T22:50:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_21.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_21.export 2024-04-24T22:50:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:50:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_22 2024-04-24T22:50:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_22 snap 2024-04-24T22:50:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T22:50:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:50:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-04-24T22:50:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T22:50:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-04-24T22:50:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T22:50:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:50:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:50:26.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:50:26.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:50:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-04-24T22:50:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T22:50:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:50:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:50:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:50:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-04-24T22:50:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:50:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:50:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:50:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:50:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:50:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:50:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:50:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:50:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:50:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-04-24T22:50:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:50:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:50:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:50:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:50:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:50:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:50:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:50:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:50:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:50:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:50:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:50:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:50:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:50:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:50:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:50:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:50:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-04-24T22:50:27.665 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-24T22:50:27.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-24T22:50:27.666 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-04-24T22:50:27.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T22:50:27.665+0000 7f15f205b700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 65011) UID: 1000 2024-04-24T22:50:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T22:50:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2024-04-24T22:50:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:50:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:50:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:50:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:50:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:50:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:50:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:50:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:50:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:50:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:50:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:50:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:50:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:50:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:50:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:50:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:50:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2024-04-24T22:50:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-04-24T22:50:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_22 2024-04-24T22:50:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:50:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:50:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:50:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:50:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:50:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:50:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:50:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:50:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:50:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:50:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:50:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:50:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:50:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:50:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:50:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:50:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:27.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:27.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_22 2024-04-24T22:50:27.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:50:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-04-24T22:50:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_22 2024-04-24T22:50:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:50:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:50:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:50:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:50:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:50:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:50:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:50:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:50:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:50:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:50:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:50:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:50:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:50:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:50:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:50:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:50:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_22 2024-04-24T22:50:28.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:50:28.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:50:28.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:50:28.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:50:28.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:50:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:50:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:50:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2024-04-24T22:50:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:50:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T22:50:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:50:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:50:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2024-04-24T22:50:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:50:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T22:50:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_22.status 2024-04-24T22:50:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_22.status 2024-04-24T22:50:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:50:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_22.status 2024-04-24T22:50:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_22.status 2024-04-24T22:50:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_22.status 2024-04-24T22:50:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:50:28.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_22.status 2024-04-24T22:50:28.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2024-04-24T22:50:28.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:28.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T22:50:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2024-04-24T22:50:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 161 2024-04-24T22:50:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_22.status 2024-04-24T22:50:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T22:50:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_22.status 2024-04-24T22:50:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2024-04-24T22:50:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:50:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=161 2024-04-24T22:50:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:28.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:50:28.667 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-24T22:50:28.667 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:50:28.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:50:28.667 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-24T22:50:28.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-24T22:50:28.668 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-24T22:50:28.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-24T22:50:28.670 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2024-04-24T22:50:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_22.status 2024-04-24T22:50:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T22:50:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_22.status 2024-04-24T22:50:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-04-24T22:50:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=89 2024-04-24T22:50:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 161 = 89 2024-04-24T22:50:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:50:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_22.status 2024-04-24T22:50:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T22:50:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_22.status 2024-04-24T22:50:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-04-24T22:50:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=89 2024-04-24T22:50:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 161 = 89 2024-04-24T22:50:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:50:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_22.status 2024-04-24T22:50:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T22:50:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_22.status 2024-04-24T22:50:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-04-24T22:50:30.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:50:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:50:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:50:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:50:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 161 = '' 2024-04-24T22:50:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T22:50:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_22.status 2024-04-24T22:50:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T22:50:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_22.status 2024-04-24T22:50:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-04-24T22:50:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=161 2024-04-24T22:50:31.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 161 = 161 2024-04-24T22:50:31.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:31.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-04-24T22:50:31.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:31.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:31.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T22:50:31.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:31.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:50:31.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:50:31.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:50:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-04-24T22:50:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_22': 2024-04-24T22:50:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:50:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:50:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:50:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dcf874ce2b 2024-04-24T22:50:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dcf874ce2b 2024-04-24T22:50:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:50:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:50:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:50:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:50:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:42:43 2024 2024-04-24T22:50:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:42:43 2024 2024-04-24T22:50:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:42:43 2024 2024-04-24T22:50:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:50:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:50:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:50:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 6ab5e540-e8f6-406b-9bdd-00cf740c3964 2024-04-24T22:50:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:50:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_22 snap 2024-04-24T22:50:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T22:50:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:50:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_22.export 2024-04-24T22:50:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_22.export 2024-04-24T22:50:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_22.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_22.export 2024-04-24T22:50:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_22.export 2024-04-24T22:50:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:50:33.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_22.export 2024-04-24T22:50:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:50:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_22.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_22.export 2024-04-24T22:50:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_22.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_22.export 2024-04-24T22:50:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:50:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_23 2024-04-24T22:50:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_23 snap 2024-04-24T22:50:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T22:50:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:50:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-04-24T22:50:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T22:50:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-04-24T22:50:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T22:50:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:50:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:50:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:50:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:50:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-04-24T22:50:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T22:50:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:50:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:50:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:50:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-04-24T22:50:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:50:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:50:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:50:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:50:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:50:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:50:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:50:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:50:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:50:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-04-24T22:50:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:50:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:50:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:50:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:50:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:50:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:50:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:50:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:50:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:50:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:50:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:50:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:50:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:50:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:50:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:50:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:50:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-04-24T22:50:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-04-24T22:50:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:36.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2024-04-24T22:50:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:50:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:50:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:50:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:50:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:50:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:50:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:50:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:50:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:50:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:50:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:50:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:50:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:50:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:50:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:50:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:50:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2024-04-24T22:50:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-04-24T22:50:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2024-04-24T22:50:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:50:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:50:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:50:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:50:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:50:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:50:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:50:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:50:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:50:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:50:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:50:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:50:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:50:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:50:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:50:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:50:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2024-04-24T22:50:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-04-24T22:50:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_23 2024-04-24T22:50:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:50:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:50:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:50:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:50:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:50:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:50:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:50:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:50:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:50:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:50:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:50:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:50:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:50:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:50:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:50:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:50:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_23 2024-04-24T22:50:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:50:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:50:36.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:50:36.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:50:36.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:50:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:50:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:50:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2024-04-24T22:50:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:50:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T22:50:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:50:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:50:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2024-04-24T22:50:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:50:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T22:50:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_23.status 2024-04-24T22:50:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_23.status 2024-04-24T22:50:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:50:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_23.status 2024-04-24T22:50:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_23.status 2024-04-24T22:50:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_23.status 2024-04-24T22:50:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:50:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_23.status 2024-04-24T22:50:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2024-04-24T22:50:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T22:50:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2024-04-24T22:50:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 163 2024-04-24T22:50:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_23.status 2024-04-24T22:50:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T22:50:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_23.status 2024-04-24T22:50:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2024-04-24T22:50:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:50:37.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=163 2024-04-24T22:50:37.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:37.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:37.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:50:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_23.status 2024-04-24T22:50:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T22:50:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_23.status 2024-04-24T22:50:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-04-24T22:50:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:50:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:50:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:50:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:50:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 163 = '' 2024-04-24T22:50:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:50:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_23.status 2024-04-24T22:50:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T22:50:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_23.status 2024-04-24T22:50:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-04-24T22:50:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:50:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:50:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:50:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:50:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 163 = '' 2024-04-24T22:50:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:50:38.413 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T22:50:38.410+0000 7fb1e9f85700 -1 reset not still connected to 0x5586e349cea0 2024-04-24T22:50:38.413 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T22:50:38.410+0000 7fb1e9f85700 -1 reset not still connected to 0x5586e5250270 2024-04-24T22:50:38.413 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T22:50:38.411+0000 7fb1e9f85700 -1 reset not still connected to 0x5586ef249930 2024-04-24T22:50:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_23.status 2024-04-24T22:50:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:38.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T22:50:38.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_23.status 2024-04-24T22:50:38.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-04-24T22:50:38.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=163 2024-04-24T22:50:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 163 = 163 2024-04-24T22:50:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-04-24T22:50:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T22:50:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:50:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:50:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:50:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-04-24T22:50:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_23': 2024-04-24T22:50:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:50:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:50:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:50:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dcf742e387 2024-04-24T22:50:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dcf742e387 2024-04-24T22:50:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:50:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:50:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:50:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:50:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:42:50 2024 2024-04-24T22:50:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:42:50 2024 2024-04-24T22:50:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:42:50 2024 2024-04-24T22:50:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:50:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:50:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:50:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: c24b92fa-65bf-4177-9441-bcf5f91f92ff 2024-04-24T22:50:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:50:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_23 snap 2024-04-24T22:50:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T22:50:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:50:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_23.export 2024-04-24T22:50:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_23.export 2024-04-24T22:50:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_23.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_23.export 2024-04-24T22:50:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_23.export 2024-04-24T22:50:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:50:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_23.export 2024-04-24T22:50:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:50:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_23.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_23.export 2024-04-24T22:50:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_23.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_23.export 2024-04-24T22:50:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:50:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_24 2024-04-24T22:50:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_24 snap 2024-04-24T22:50:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T22:50:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:50:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-04-24T22:50:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T22:50:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-04-24T22:50:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T22:50:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:50:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:50:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:50:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:50:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-04-24T22:50:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T22:50:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:50:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:50:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:50:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-04-24T22:50:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:50:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:50:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:50:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:50:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:50:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:50:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:50:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:50:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:50:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-04-24T22:50:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:50:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:50:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:50:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:50:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:50:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:50:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:50:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:50:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:50:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:50:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:50:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:50:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:50:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:50:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:50:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:50:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-04-24T22:50:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-04-24T22:50:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:43.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:43.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2024-04-24T22:50:43.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:50:43.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:43.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:50:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:50:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:50:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:50:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:50:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:50:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:50:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:50:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:50:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:50:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:50:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:50:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:50:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:50:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:50:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2024-04-24T22:50:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-04-24T22:50:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2024-04-24T22:50:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:50:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:50:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:50:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:50:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:50:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:50:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:50:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:50:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:50:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:50:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:50:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:50:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:50:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:50:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:50:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:50:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2024-04-24T22:50:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-04-24T22:50:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_24 2024-04-24T22:50:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:50:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:50:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:50:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:50:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:50:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:50:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:50:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:50:43.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:50:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:50:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:50:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:50:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:50:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:50:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:50:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:50:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_24 2024-04-24T22:50:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:50:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:50:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:50:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:50:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:50:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:50:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:50:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2024-04-24T22:50:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:50:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T22:50:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:50:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:50:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2024-04-24T22:50:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:50:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T22:50:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_24.status 2024-04-24T22:50:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_24.status 2024-04-24T22:50:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:50:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_24.status 2024-04-24T22:50:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_24.status 2024-04-24T22:50:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_24.status 2024-04-24T22:50:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:50:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_24.status 2024-04-24T22:50:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2024-04-24T22:50:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T22:50:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2024-04-24T22:50:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 165 2024-04-24T22:50:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_24.status 2024-04-24T22:50:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T22:50:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_24.status 2024-04-24T22:50:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2024-04-24T22:50:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:50:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=165 2024-04-24T22:50:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:50:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_24.status 2024-04-24T22:50:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T22:50:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_24.status 2024-04-24T22:50:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-04-24T22:50:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:44.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:44.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:44.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:50:44.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:50:44.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:50:44.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:50:44.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 165 = '' 2024-04-24T22:50:44.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:44.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:50:44.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_24.status 2024-04-24T22:50:44.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T22:50:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_24.status 2024-04-24T22:50:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-04-24T22:50:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:50:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:50:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:50:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:50:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 165 = '' 2024-04-24T22:50:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:50:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_24.status 2024-04-24T22:50:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T22:50:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_24.status 2024-04-24T22:50:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-04-24T22:50:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=165 2024-04-24T22:50:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 165 = 165 2024-04-24T22:50:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-04-24T22:50:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:45.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T22:50:45.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:45.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:50:45.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:50:45.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:50:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-04-24T22:50:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_24': 2024-04-24T22:50:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:50:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:50:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:50:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc4301a7c1 2024-04-24T22:50:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc4301a7c1 2024-04-24T22:50:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:50:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:50:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:50:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:50:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:43:00 2024 2024-04-24T22:50:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:43:00 2024 2024-04-24T22:50:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:43:00 2024 2024-04-24T22:50:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:50:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:50:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:50:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: f25eb5c2-2f4d-4c6e-9c1e-1ee69ca3048a 2024-04-24T22:50:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:50:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_24 snap 2024-04-24T22:50:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T22:50:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:50:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_24.export 2024-04-24T22:50:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_24.export 2024-04-24T22:50:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_24.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_24.export 2024-04-24T22:50:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_24.export 2024-04-24T22:50:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:50:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_24.export 2024-04-24T22:50:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:50:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_24.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_24.export 2024-04-24T22:50:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_24.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_24.export 2024-04-24T22:50:47.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:47.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:50:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_25 2024-04-24T22:50:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_25 snap 2024-04-24T22:50:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T22:50:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:50:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-04-24T22:50:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T22:50:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-04-24T22:50:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T22:50:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:50:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:50:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:50:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:50:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-04-24T22:50:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T22:50:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:50:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:50:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:50:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-04-24T22:50:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:50:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:50:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:50:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:50:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:50:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:50:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:50:50.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:50:50.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:50:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-04-24T22:50:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:50:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:50:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:50:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:50:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:50:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:50:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:50:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:50:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:50:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:50:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:50:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:50:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:50:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:50:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:50:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:50:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-04-24T22:50:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-04-24T22:50:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2024-04-24T22:50:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:50:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:50:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:50:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:50:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:50:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:50:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:50:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:50:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:50:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:50:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:50:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:50:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:50:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:50:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:50:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:50:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2024-04-24T22:50:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-04-24T22:50:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_25 2024-04-24T22:50:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:50:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:50:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:50:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:50:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:50:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:50:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:50:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:50:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:50:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:50:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:50:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:50:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:50:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:50:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:50:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:50:50.539 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7726153878379365) > weight (0.25) 2024-04-24T22:50:50.539 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7116390702555553) > weight (0.25) 2024-04-24T22:50:50.539 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4557221346632556) > weight (0.25) 2024-04-24T22:50:50.539 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.31828144379419576) > weight (0.25) 2024-04-24T22:50:50.539 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2024-04-24T22:50:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_25 2024-04-24T22:50:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-04-24T22:50:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_25 2024-04-24T22:50:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:50:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:50:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:50:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:50:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:50:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:50:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:50:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:50:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:50:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:50:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:50:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:50:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:50:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:50:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:50:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:50:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_25 2024-04-24T22:50:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:50:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:50:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:50:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:50:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:50:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:50:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:50:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2024-04-24T22:50:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:50:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T22:50:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:50:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:50:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2024-04-24T22:50:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:50:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T22:50:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_25.status 2024-04-24T22:50:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_25.status 2024-04-24T22:50:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:50:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_25.status 2024-04-24T22:50:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_25.status 2024-04-24T22:50:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_25.status 2024-04-24T22:50:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:50:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_25.status 2024-04-24T22:50:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2024-04-24T22:50:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T22:50:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2024-04-24T22:50:51.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 167 2024-04-24T22:50:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_25.status 2024-04-24T22:50:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T22:50:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_25.status 2024-04-24T22:50:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2024-04-24T22:50:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:50:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=167 2024-04-24T22:50:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:50:51.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_25.status 2024-04-24T22:50:51.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:51.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:51.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T22:50:51.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_25.status 2024-04-24T22:50:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-04-24T22:50:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:51.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:51.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:51.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:50:51.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:50:51.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:50:51.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:50:51.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 167 = '' 2024-04-24T22:50:51.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:51.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:50:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_25.status 2024-04-24T22:50:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T22:50:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_25.status 2024-04-24T22:50:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-04-24T22:50:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:50:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:50:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:50:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:50:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 167 = '' 2024-04-24T22:50:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:50:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_25.status 2024-04-24T22:50:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T22:50:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_25.status 2024-04-24T22:50:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-04-24T22:50:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:52.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=167 2024-04-24T22:50:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 167 = 167 2024-04-24T22:50:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-04-24T22:50:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T22:50:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:50:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:50:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:50:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-04-24T22:50:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_25': 2024-04-24T22:50:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:50:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:50:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:50:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc63303844 2024-04-24T22:50:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc63303844 2024-04-24T22:50:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:50:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:50:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:50:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:50:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:43:13 2024 2024-04-24T22:50:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:43:13 2024 2024-04-24T22:50:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:43:13 2024 2024-04-24T22:50:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:50:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:50:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:50:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: a6830866-682c-4c96-8b3b-75517871f35d 2024-04-24T22:50:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:50:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_25 snap 2024-04-24T22:50:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T22:50:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:50:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_25.export 2024-04-24T22:50:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_25.export 2024-04-24T22:50:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_25.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_25.export 2024-04-24T22:50:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_25.export 2024-04-24T22:50:54.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:50:54.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_25.export 2024-04-24T22:50:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:50:54.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_25.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_25.export 2024-04-24T22:50:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_25.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_25.export 2024-04-24T22:50:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:50:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_26 2024-04-24T22:50:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_26 snap 2024-04-24T22:50:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T22:50:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:50:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-04-24T22:50:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T22:50:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-04-24T22:50:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T22:50:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:50:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:50:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:50:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:50:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-04-24T22:50:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T22:50:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:50:57.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:50:57.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:50:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-04-24T22:50:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:50:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:50:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:50:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:50:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:50:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:50:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:50:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:50:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:50:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-04-24T22:50:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:50:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:50:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:50:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:50:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:50:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:50:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:50:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:50:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:50:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:50:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:50:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:50:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:50:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:50:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:50:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:50:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:50:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-04-24T22:50:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-04-24T22:50:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2024-04-24T22:50:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:50:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:50:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:50:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:50:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:50:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:50:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:50:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:50:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:50:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:50:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:50:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:50:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:50:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:50:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:50:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:50:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:50:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2024-04-24T22:50:57.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-04-24T22:50:57.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:57.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2024-04-24T22:50:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:50:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:50:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:50:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:50:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:50:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:50:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:50:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:50:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:50:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:50:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:50:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:50:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:50:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:50:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:50:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:50:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:50:57.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2024-04-24T22:50:57.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-04-24T22:50:57.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:50:57.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:50:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:50:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_26 2024-04-24T22:50:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:50:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:50:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:50:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:50:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:50:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:50:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:50:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:50:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:50:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:50:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:50:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:50:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:50:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:50:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:50:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:50:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:50:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:50:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:50:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:50:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:50:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:50:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:50:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:50:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:50:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:50:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:50:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:50:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:50:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:50:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_26 2024-04-24T22:50:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:50:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:50:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:50:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:50:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:50:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:50:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:50:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2024-04-24T22:50:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:50:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T22:50:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:50:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:50:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2024-04-24T22:50:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:50:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T22:50:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_26.status 2024-04-24T22:50:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_26.status 2024-04-24T22:50:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:50:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_26.status 2024-04-24T22:50:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_26.status 2024-04-24T22:50:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_26.status 2024-04-24T22:50:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:50:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_26.status 2024-04-24T22:50:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2024-04-24T22:50:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T22:50:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2024-04-24T22:50:58.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 169 2024-04-24T22:50:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_26.status 2024-04-24T22:50:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:50:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T22:50:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_26.status 2024-04-24T22:50:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2024-04-24T22:50:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:50:58.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=169 2024-04-24T22:50:58.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:58.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:58.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:50:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_26.status 2024-04-24T22:50:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T22:50:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_26.status 2024-04-24T22:50:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-04-24T22:50:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:50:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:50:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:50:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:50:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 169 = '' 2024-04-24T22:50:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:50:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_26.status 2024-04-24T22:50:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T22:50:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_26.status 2024-04-24T22:50:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-04-24T22:50:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:50:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:50:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:50:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:50:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:50:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 169 = '' 2024-04-24T22:50:59.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:50:59.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:50:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_26.status 2024-04-24T22:50:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T22:50:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_26.status 2024-04-24T22:50:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-04-24T22:50:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:50:59.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:50:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=169 2024-04-24T22:50:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 169 = 169 2024-04-24T22:50:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:50:59.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-04-24T22:50:59.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:50:59.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:50:59.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T22:50:59.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:50:59.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:50:59.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:50:59.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:51:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-04-24T22:51:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_26': 2024-04-24T22:51:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:51:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:51:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:51:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dce944fade 2024-04-24T22:51:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dce944fade 2024-04-24T22:51:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:51:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:51:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:51:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:51:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:43:28 2024 2024-04-24T22:51:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:43:28 2024 2024-04-24T22:51:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:43:28 2024 2024-04-24T22:51:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:51:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:51:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:51:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: f38aef72-1054-47c3-917d-19c1b6627e53 2024-04-24T22:51:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:51:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_26 snap 2024-04-24T22:51:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T22:51:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:51:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_26.export 2024-04-24T22:51:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_26.export 2024-04-24T22:51:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_26.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_26.export 2024-04-24T22:51:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_26.export 2024-04-24T22:51:01.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:51:01.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_26.export 2024-04-24T22:51:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:51:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_26.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_26.export 2024-04-24T22:51:01.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_26.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_26.export 2024-04-24T22:51:01.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:01.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:51:01.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_27 2024-04-24T22:51:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_27 snap 2024-04-24T22:51:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T22:51:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:51:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-04-24T22:51:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T22:51:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-04-24T22:51:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T22:51:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:51:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:51:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:51:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:51:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-04-24T22:51:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T22:51:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:51:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:51:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:51:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-04-24T22:51:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:51:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:51:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:51:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:51:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:51:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:51:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:51:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:51:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:51:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-04-24T22:51:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:51:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:51:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:51:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:51:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:51:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:51:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:51:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:51:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:51:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:51:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:51:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:51:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:51:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:51:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:51:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:51:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:04.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-04-24T22:51:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-04-24T22:51:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2024-04-24T22:51:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:51:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:51:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:51:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:51:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:51:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:51:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:51:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:51:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:51:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:51:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:51:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:51:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:51:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:51:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:51:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:51:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2024-04-24T22:51:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-04-24T22:51:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2024-04-24T22:51:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:51:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:51:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:51:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:51:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:51:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:51:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:51:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:51:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:51:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2024-04-24T22:51:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-04-24T22:51:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_27 2024-04-24T22:51:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:51:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:51:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:51:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:51:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:51:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:51:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:51:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:51:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:51:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:51:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:51:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:51:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:51:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:51:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:51:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:51:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_27 2024-04-24T22:51:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:51:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:51:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:51:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:51:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:51:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:51:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:51:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2024-04-24T22:51:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:51:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T22:51:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:51:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:51:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2024-04-24T22:51:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:51:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T22:51:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_27.status 2024-04-24T22:51:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_27.status 2024-04-24T22:51:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:51:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_27.status 2024-04-24T22:51:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_27.status 2024-04-24T22:51:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_27.status 2024-04-24T22:51:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:51:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_27.status 2024-04-24T22:51:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2024-04-24T22:51:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T22:51:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2024-04-24T22:51:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 171 2024-04-24T22:51:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_27.status 2024-04-24T22:51:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T22:51:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_27.status 2024-04-24T22:51:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2024-04-24T22:51:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:51:05.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=171 2024-04-24T22:51:05.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:51:05.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:05.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:51:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_27.status 2024-04-24T22:51:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T22:51:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_27.status 2024-04-24T22:51:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-04-24T22:51:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:51:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:51:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:51:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:51:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:51:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 171 = '' 2024-04-24T22:51:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:51:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_27.status 2024-04-24T22:51:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T22:51:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_27.status 2024-04-24T22:51:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-04-24T22:51:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:51:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:51:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:51:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:51:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:51:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 171 = '' 2024-04-24T22:51:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:51:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_27.status 2024-04-24T22:51:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T22:51:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_27.status 2024-04-24T22:51:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-04-24T22:51:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=171 2024-04-24T22:51:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 171 = 171 2024-04-24T22:51:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-04-24T22:51:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T22:51:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:51:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:51:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:51:07.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-04-24T22:51:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_27': 2024-04-24T22:51:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:51:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:51:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:51:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc872757c6 2024-04-24T22:51:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc872757c6 2024-04-24T22:51:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:51:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:51:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:51:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:51:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:43:42 2024 2024-04-24T22:51:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:43:42 2024 2024-04-24T22:51:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:43:42 2024 2024-04-24T22:51:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:51:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:51:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:51:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 1143f5a5-897d-4fc6-98d2-c436f71170b7 2024-04-24T22:51:07.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:51:07.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:07.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_27 snap 2024-04-24T22:51:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T22:51:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:51:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_27.export 2024-04-24T22:51:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_27.export 2024-04-24T22:51:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_27.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_27.export 2024-04-24T22:51:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_27.export 2024-04-24T22:51:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:51:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_27.export 2024-04-24T22:51:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:51:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_27.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_27.export 2024-04-24T22:51:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_27.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_27.export 2024-04-24T22:51:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:51:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_28 2024-04-24T22:51:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_28 snap 2024-04-24T22:51:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T22:51:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:51:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-04-24T22:51:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T22:51:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-04-24T22:51:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T22:51:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:51:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:51:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:51:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:51:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-04-24T22:51:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T22:51:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:51:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:51:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:51:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-04-24T22:51:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:51:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:51:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:51:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:51:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:51:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:51:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:51:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:51:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:51:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-04-24T22:51:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:51:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:51:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:51:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:51:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:51:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:51:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:51:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:51:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:51:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:51:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:51:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:51:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:51:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:51:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:51:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:51:11.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:11.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:11.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-04-24T22:51:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-04-24T22:51:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2024-04-24T22:51:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:51:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:51:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:51:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:51:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:51:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:51:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:51:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:51:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:51:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:51:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:51:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:51:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:51:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:51:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:51:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:51:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2024-04-24T22:51:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-04-24T22:51:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_28 2024-04-24T22:51:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:51:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:51:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:51:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:51:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:51:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:51:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:51:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:51:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:51:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:51:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:51:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:51:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:51:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:51:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:51:11.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:51:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:11.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_28 2024-04-24T22:51:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-04-24T22:51:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:11.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_28 2024-04-24T22:51:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:51:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:51:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:51:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:51:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:51:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:51:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:51:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:51:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:51:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:51:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:51:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:51:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:51:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:51:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:51:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:51:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_28 2024-04-24T22:51:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:51:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:51:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:51:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:51:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:51:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:51:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:51:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2024-04-24T22:51:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:51:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T22:51:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:51:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:51:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2024-04-24T22:51:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:51:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T22:51:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_28.status 2024-04-24T22:51:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_28.status 2024-04-24T22:51:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:51:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_28.status 2024-04-24T22:51:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_28.status 2024-04-24T22:51:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_28.status 2024-04-24T22:51:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:51:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_28.status 2024-04-24T22:51:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2024-04-24T22:51:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T22:51:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2024-04-24T22:51:12.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 173 2024-04-24T22:51:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_28.status 2024-04-24T22:51:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T22:51:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_28.status 2024-04-24T22:51:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2024-04-24T22:51:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:12.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:51:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=173 2024-04-24T22:51:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:51:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:51:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_28.status 2024-04-24T22:51:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T22:51:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_28.status 2024-04-24T22:51:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-04-24T22:51:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:12.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:51:12.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:51:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:51:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:51:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:51:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 173 = '' 2024-04-24T22:51:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:51:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_28.status 2024-04-24T22:51:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T22:51:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_28.status 2024-04-24T22:51:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-04-24T22:51:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:51:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:51:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:51:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:51:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:51:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 173 = '' 2024-04-24T22:51:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:51:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_28.status 2024-04-24T22:51:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T22:51:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_28.status 2024-04-24T22:51:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-04-24T22:51:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=173 2024-04-24T22:51:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 173 = 173 2024-04-24T22:51:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-04-24T22:51:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T22:51:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:51:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:51:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:51:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-04-24T22:51:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_28': 2024-04-24T22:51:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:51:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:51:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:51:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc9faf5725 2024-04-24T22:51:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc9faf5725 2024-04-24T22:51:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:51:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:51:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:51:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:51:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:43:54 2024 2024-04-24T22:51:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:43:54 2024 2024-04-24T22:51:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:43:54 2024 2024-04-24T22:51:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:51:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:51:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:51:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 0f1733b8-4c9d-4886-af25-8d2e8af19dd6 2024-04-24T22:51:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:51:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_28 snap 2024-04-24T22:51:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T22:51:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:51:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_28.export 2024-04-24T22:51:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_28.export 2024-04-24T22:51:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_28.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_28.export 2024-04-24T22:51:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_28.export 2024-04-24T22:51:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:51:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_28.export 2024-04-24T22:51:15.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:51:15.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_28.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_28.export 2024-04-24T22:51:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_28.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_28.export 2024-04-24T22:51:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:51:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_29 2024-04-24T22:51:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_29 snap 2024-04-24T22:51:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T22:51:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:51:16.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-04-24T22:51:16.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:16.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:16.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T22:51:16.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-04-24T22:51:16.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:16.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:16.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T22:51:16.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:51:16.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:51:16.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:51:16.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:51:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-04-24T22:51:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T22:51:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:51:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:51:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:51:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-04-24T22:51:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:51:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:51:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:51:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:51:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:51:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:51:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:51:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:51:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:51:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-04-24T22:51:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:51:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:51:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:51:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:51:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:51:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:51:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:51:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:51:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:51:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:51:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:51:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:51:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:51:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:51:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:51:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:51:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-04-24T22:51:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T22:51:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2024-04-24T22:51:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:51:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:51:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:51:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:51:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:51:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:51:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:51:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:51:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:51:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:51:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:51:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:51:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:51:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:51:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:51:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:51:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2024-04-24T22:51:17.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T22:51:17.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:17.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2024-04-24T22:51:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:51:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:51:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:51:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:51:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:51:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:51:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:51:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:51:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:51:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:51:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:51:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:51:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:51:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:51:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:51:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:51:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2024-04-24T22:51:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-04-24T22:51:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_29 2024-04-24T22:51:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:51:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:51:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:51:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:51:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:51:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:51:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:51:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:51:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:51:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:51:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:51:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:51:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:51:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:51:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:51:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:51:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_29 2024-04-24T22:51:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:51:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:51:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:51:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:51:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:51:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:51:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:51:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2024-04-24T22:51:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:51:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T22:51:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:51:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:51:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2024-04-24T22:51:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:51:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T22:51:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_29.status 2024-04-24T22:51:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_29.status 2024-04-24T22:51:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:51:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_29.status 2024-04-24T22:51:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_29.status 2024-04-24T22:51:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_29.status 2024-04-24T22:51:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:51:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_29.status 2024-04-24T22:51:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2024-04-24T22:51:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T22:51:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2024-04-24T22:51:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 175 2024-04-24T22:51:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_29.status 2024-04-24T22:51:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T22:51:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_29.status 2024-04-24T22:51:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2024-04-24T22:51:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:51:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=175 2024-04-24T22:51:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:51:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:51:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_29.status 2024-04-24T22:51:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T22:51:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_29.status 2024-04-24T22:51:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-04-24T22:51:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=96 2024-04-24T22:51:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 175 = 96 2024-04-24T22:51:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:51:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_29.status 2024-04-24T22:51:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T22:51:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_29.status 2024-04-24T22:51:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-04-24T22:51:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=96 2024-04-24T22:51:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 175 = 96 2024-04-24T22:51:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:51:20.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_29.status 2024-04-24T22:51:20.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:20.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:20.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T22:51:20.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_29.status 2024-04-24T22:51:20.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-04-24T22:51:20.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:51:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:51:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:51:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:51:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:51:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 175 = '' 2024-04-24T22:51:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T22:51:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_29.status 2024-04-24T22:51:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T22:51:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_29.status 2024-04-24T22:51:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-04-24T22:51:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=175 2024-04-24T22:51:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 175 = 175 2024-04-24T22:51:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-04-24T22:51:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T22:51:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:51:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:51:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:51:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-04-24T22:51:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_29': 2024-04-24T22:51:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:51:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:51:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:51:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc115321ab 2024-04-24T22:51:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc115321ab 2024-04-24T22:51:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:51:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:51:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:51:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:51:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:44:00 2024 2024-04-24T22:51:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:44:00 2024 2024-04-24T22:51:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:44:00 2024 2024-04-24T22:51:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:51:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:51:22.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:51:22.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 171b9277-b7b3-4a7f-ae2b-c3d8aca7eadf 2024-04-24T22:51:22.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:51:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_29 snap 2024-04-24T22:51:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T22:51:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:51:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_29.export 2024-04-24T22:51:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_29.export 2024-04-24T22:51:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_29.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_29.export 2024-04-24T22:51:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_29.export 2024-04-24T22:51:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:51:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_29.export 2024-04-24T22:51:23.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:51:23.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_29.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_29.export 2024-04-24T22:51:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_29.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_29.export 2024-04-24T22:51:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:51:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_30 2024-04-24T22:51:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_30 snap 2024-04-24T22:51:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T22:51:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:51:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-04-24T22:51:25.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T22:51:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-04-24T22:51:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T22:51:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:51:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:51:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:51:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:51:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-04-24T22:51:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T22:51:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:51:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:51:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:51:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-04-24T22:51:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:51:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:51:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:51:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:51:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:51:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:51:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:51:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:51:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:51:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-04-24T22:51:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:51:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:51:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:51:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:51:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:51:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:51:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:51:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:51:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:51:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:51:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:51:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:51:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:51:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:51:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:51:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:51:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-04-24T22:51:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-24T22:51:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2024-04-24T22:51:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:51:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:51:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:51:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:51:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:51:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:51:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:51:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:51:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:51:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:51:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:51:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:51:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:51:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:51:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:51:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:51:26.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:26.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:26.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:26.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2024-04-24T22:51:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-24T22:51:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2024-04-24T22:51:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:51:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:51:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:51:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:51:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:51:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:51:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:51:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:51:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:51:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:51:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:51:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:51:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:51:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:51:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:51:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:51:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2024-04-24T22:51:26.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-04-24T22:51:26.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:26.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:26.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_30 2024-04-24T22:51:26.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:51:26.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:26.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:26.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:51:26.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:51:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:51:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:51:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:51:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:51:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:51:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:51:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:51:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:51:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:51:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:51:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:51:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:51:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:51:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_30 2024-04-24T22:51:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:51:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:51:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:51:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:51:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:51:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:51:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:51:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2024-04-24T22:51:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:51:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T22:51:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:51:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:51:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2024-04-24T22:51:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:51:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T22:51:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_30.status 2024-04-24T22:51:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_30.status 2024-04-24T22:51:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:51:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_30.status 2024-04-24T22:51:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_30.status 2024-04-24T22:51:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_30.status 2024-04-24T22:51:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:51:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_30.status 2024-04-24T22:51:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2024-04-24T22:51:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T22:51:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2024-04-24T22:51:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 177 2024-04-24T22:51:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_30.status 2024-04-24T22:51:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T22:51:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_30.status 2024-04-24T22:51:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2024-04-24T22:51:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:51:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=177 2024-04-24T22:51:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:51:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:51:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_30.status 2024-04-24T22:51:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T22:51:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_30.status 2024-04-24T22:51:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-04-24T22:51:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:27.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:51:27.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:51:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:51:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:51:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:51:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 177 = '' 2024-04-24T22:51:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:51:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_30.status 2024-04-24T22:51:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T22:51:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_30.status 2024-04-24T22:51:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-04-24T22:51:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:51:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:51:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:51:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:51:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:51:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 177 = '' 2024-04-24T22:51:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:51:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_30.status 2024-04-24T22:51:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T22:51:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_30.status 2024-04-24T22:51:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-04-24T22:51:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=177 2024-04-24T22:51:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 177 = 177 2024-04-24T22:51:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-04-24T22:51:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T22:51:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:51:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:51:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:51:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-04-24T22:51:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_30': 2024-04-24T22:51:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:51:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:51:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:51:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc2dda2b91 2024-04-24T22:51:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc2dda2b91 2024-04-24T22:51:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:51:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:51:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:51:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:51:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:44:13 2024 2024-04-24T22:51:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:44:13 2024 2024-04-24T22:51:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:44:13 2024 2024-04-24T22:51:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:51:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:51:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:51:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 4cd18b79-d4e1-402a-89e5-a62956f139ed 2024-04-24T22:51:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:51:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_30 snap 2024-04-24T22:51:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T22:51:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:51:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_30.export 2024-04-24T22:51:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_30.export 2024-04-24T22:51:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_30.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_30.export 2024-04-24T22:51:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_30.export 2024-04-24T22:51:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:51:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_30.export 2024-04-24T22:51:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:51:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_30.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_30.export 2024-04-24T22:51:30.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_30.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_30.export 2024-04-24T22:51:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:51:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_31 2024-04-24T22:51:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_31 snap 2024-04-24T22:51:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T22:51:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:30.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:51:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-04-24T22:51:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T22:51:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-04-24T22:51:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T22:51:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:51:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:51:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:51:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:51:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-04-24T22:51:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T22:51:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:51:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:51:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:51:33.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-04-24T22:51:33.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:51:33.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:51:33.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:33.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:51:33.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:51:33.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:51:33.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:51:33.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:51:33.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:51:33.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:51:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-04-24T22:51:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:51:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:51:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:51:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:51:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:51:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:51:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:51:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:51:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:51:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:51:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:51:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:51:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:51:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:51:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:51:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:51:33.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:33.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:33.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:33.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-04-24T22:51:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-04-24T22:51:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2024-04-24T22:51:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:51:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:51:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:51:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:51:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:51:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:51:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:51:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:51:33.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:33.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:33.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:33.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2024-04-24T22:51:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-04-24T22:51:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_31 2024-04-24T22:51:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:51:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:51:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:51:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:51:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:51:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:51:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:51:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:51:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:51:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:51:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:51:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:51:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:51:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:33.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:33.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:51:33.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:51:33.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:51:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_31 2024-04-24T22:51:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-04-24T22:51:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_31 2024-04-24T22:51:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:51:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:51:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:51:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:51:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:51:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:51:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:51:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:51:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:51:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:51:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:51:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:51:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:51:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:51:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:51:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:51:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_31 2024-04-24T22:51:33.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:51:33.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:51:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:51:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:51:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:51:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:51:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:51:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2024-04-24T22:51:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:51:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T22:51:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:51:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:51:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2024-04-24T22:51:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:51:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T22:51:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_31.status 2024-04-24T22:51:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_31.status 2024-04-24T22:51:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:51:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_31.status 2024-04-24T22:51:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_31.status 2024-04-24T22:51:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_31.status 2024-04-24T22:51:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:51:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_31.status 2024-04-24T22:51:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2024-04-24T22:51:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T22:51:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2024-04-24T22:51:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 179 2024-04-24T22:51:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_31.status 2024-04-24T22:51:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T22:51:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_31.status 2024-04-24T22:51:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2024-04-24T22:51:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:51:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=179 2024-04-24T22:51:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:51:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:51:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_31.status 2024-04-24T22:51:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T22:51:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_31.status 2024-04-24T22:51:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-04-24T22:51:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:34.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:51:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:51:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:51:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:51:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:51:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 179 = '' 2024-04-24T22:51:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:51:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_31.status 2024-04-24T22:51:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T22:51:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_31.status 2024-04-24T22:51:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-04-24T22:51:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:51:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:51:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:51:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:51:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:51:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 179 = '' 2024-04-24T22:51:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:51:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_31.status 2024-04-24T22:51:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T22:51:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_31.status 2024-04-24T22:51:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-04-24T22:51:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:35.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=179 2024-04-24T22:51:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 179 = 179 2024-04-24T22:51:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-04-24T22:51:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T22:51:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:51:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:51:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:51:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-04-24T22:51:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_31': 2024-04-24T22:51:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:51:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:51:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:51:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc75191572 2024-04-24T22:51:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc75191572 2024-04-24T22:51:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:51:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:51:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:51:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:51:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:44:19 2024 2024-04-24T22:51:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:44:19 2024 2024-04-24T22:51:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:44:19 2024 2024-04-24T22:51:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:51:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:51:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:51:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: e73f5c22-d59e-4e21-8ef5-f803b29356cb 2024-04-24T22:51:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:51:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_31 snap 2024-04-24T22:51:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T22:51:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:51:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_31.export 2024-04-24T22:51:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_31.export 2024-04-24T22:51:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_31.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_31.export 2024-04-24T22:51:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_31.export 2024-04-24T22:51:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:51:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_31.export 2024-04-24T22:51:37.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:51:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_31.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_31.export 2024-04-24T22:51:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_31.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_31.export 2024-04-24T22:51:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:51:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_32 2024-04-24T22:51:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_32 snap 2024-04-24T22:51:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T22:51:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:51:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-04-24T22:51:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T22:51:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-04-24T22:51:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T22:51:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:51:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:51:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:51:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:51:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-04-24T22:51:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:40.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T22:51:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:51:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:51:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:51:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-04-24T22:51:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:51:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:51:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:51:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:51:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:51:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:51:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:51:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:51:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:51:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-04-24T22:51:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:51:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:51:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:51:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:51:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:51:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:51:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:51:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:51:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:51:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:51:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:51:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:51:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:51:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:51:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:51:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:51:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-04-24T22:51:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-04-24T22:51:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2024-04-24T22:51:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:51:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:51:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:51:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:51:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:51:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:51:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:51:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:51:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:51:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:51:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:51:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:51:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:51:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:51:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:51:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:51:40.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:40.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:40.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:40.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2024-04-24T22:51:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-04-24T22:51:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_32 2024-04-24T22:51:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:51:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:51:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:51:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:51:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:51:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:51:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:51:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:51:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:51:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:51:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:51:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:51:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:51:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:51:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:51:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:51:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_32 2024-04-24T22:51:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-04-24T22:51:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_32 2024-04-24T22:51:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:51:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:51:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:51:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:51:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:51:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:51:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:51:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:51:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:51:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:51:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:51:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:51:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:51:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:51:40.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:51:40.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:51:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_32 2024-04-24T22:51:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:51:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:51:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:51:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:51:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:51:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:51:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:51:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2024-04-24T22:51:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:51:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T22:51:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:51:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:51:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2024-04-24T22:51:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:51:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T22:51:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_32.status 2024-04-24T22:51:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_32.status 2024-04-24T22:51:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:51:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_32.status 2024-04-24T22:51:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_32.status 2024-04-24T22:51:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_32.status 2024-04-24T22:51:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:51:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_32.status 2024-04-24T22:51:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2024-04-24T22:51:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T22:51:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2024-04-24T22:51:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 181 2024-04-24T22:51:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_32.status 2024-04-24T22:51:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T22:51:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_32.status 2024-04-24T22:51:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2024-04-24T22:51:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:51:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=181 2024-04-24T22:51:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:51:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:51:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_32.status 2024-04-24T22:51:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T22:51:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_32.status 2024-04-24T22:51:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-04-24T22:51:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:51:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:51:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:51:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:51:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:51:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 181 = '' 2024-04-24T22:51:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:51:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_32.status 2024-04-24T22:51:42.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:42.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:42.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T22:51:42.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_32.status 2024-04-24T22:51:42.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-04-24T22:51:42.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:51:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:51:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:51:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:51:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:51:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 181 = '' 2024-04-24T22:51:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:51:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_32.status 2024-04-24T22:51:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T22:51:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_32.status 2024-04-24T22:51:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-04-24T22:51:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=181 2024-04-24T22:51:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 181 = 181 2024-04-24T22:51:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-04-24T22:51:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T22:51:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:51:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:51:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:51:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-04-24T22:51:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_32': 2024-04-24T22:51:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:51:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:51:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:51:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dcc81c8a56 2024-04-24T22:51:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dcc81c8a56 2024-04-24T22:51:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:51:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:51:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:51:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:51:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:44:32 2024 2024-04-24T22:51:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:44:32 2024 2024-04-24T22:51:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:44:32 2024 2024-04-24T22:51:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:51:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:51:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:51:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: f185f517-ef74-4863-98b2-5700a59468b6 2024-04-24T22:51:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:51:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_32 snap 2024-04-24T22:51:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T22:51:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:51:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_32.export 2024-04-24T22:51:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_32.export 2024-04-24T22:51:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_32.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_32.export 2024-04-24T22:51:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_32.export 2024-04-24T22:51:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:51:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_32.export 2024-04-24T22:51:44.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:51:44.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_32.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_32.export 2024-04-24T22:51:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_32.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_32.export 2024-04-24T22:51:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:51:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_33 2024-04-24T22:51:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_33 snap 2024-04-24T22:51:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T22:51:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:51:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-04-24T22:51:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T22:51:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-04-24T22:51:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T22:51:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:51:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:51:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:51:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:51:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-04-24T22:51:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T22:51:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:51:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:51:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:51:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-04-24T22:51:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:51:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:51:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:51:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:51:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:51:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:51:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:51:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:51:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:51:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-04-24T22:51:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:51:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:51:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:51:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:51:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:51:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:51:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:51:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:51:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-04-24T22:51:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T22:51:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:47.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-04-24T22:51:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:51:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:51:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:51:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:51:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:51:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:51:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:51:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:51:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:51:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:51:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:51:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:51:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:51:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:51:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:51:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:51:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-04-24T22:51:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T22:51:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2024-04-24T22:51:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:51:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:51:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:51:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:51:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:51:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:51:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:51:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:51:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:51:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:51:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:51:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:51:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:51:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:51:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:51:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:51:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2024-04-24T22:51:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-04-24T22:51:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:47.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:47.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2024-04-24T22:51:47.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:51:47.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:47.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:47.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:51:47.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:51:47.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:47.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:51:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:51:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:51:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:51:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:51:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:51:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:51:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:51:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:51:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:51:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:51:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:51:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:51:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2024-04-24T22:51:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:51:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:51:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:51:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:51:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:51:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:51:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:51:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2024-04-24T22:51:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:51:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T22:51:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:51:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:51:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2024-04-24T22:51:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:51:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T22:51:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_33.status 2024-04-24T22:51:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_33.status 2024-04-24T22:51:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:51:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_33.status 2024-04-24T22:51:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_33.status 2024-04-24T22:51:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_33.status 2024-04-24T22:51:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:51:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_33.status 2024-04-24T22:51:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2024-04-24T22:51:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T22:51:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2024-04-24T22:51:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 183 2024-04-24T22:51:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_33.status 2024-04-24T22:51:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T22:51:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_33.status 2024-04-24T22:51:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2024-04-24T22:51:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:51:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=183 2024-04-24T22:51:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:51:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:51:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_33.status 2024-04-24T22:51:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T22:51:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_33.status 2024-04-24T22:51:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-04-24T22:51:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:51:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:51:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:51:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:51:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:51:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 183 = '' 2024-04-24T22:51:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:51:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_33.status 2024-04-24T22:51:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T22:51:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_33.status 2024-04-24T22:51:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-04-24T22:51:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:51:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:49.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:51:49.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:51:49.039 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:51:49.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:51:49.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 183 = '' 2024-04-24T22:51:49.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:49.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:51:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_33.status 2024-04-24T22:51:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T22:51:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_33.status 2024-04-24T22:51:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-04-24T22:51:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=183 2024-04-24T22:51:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 183 = 183 2024-04-24T22:51:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2024-04-24T22:51:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T22:51:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:51:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:51:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:51:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2024-04-24T22:51:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_33': 2024-04-24T22:51:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:51:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:51:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:51:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc713f194c 2024-04-24T22:51:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc713f194c 2024-04-24T22:51:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:51:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:51:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:51:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:51:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:44:39 2024 2024-04-24T22:51:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:44:39 2024 2024-04-24T22:51:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:44:39 2024 2024-04-24T22:51:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:51:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:51:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:51:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 9907f064-0835-4f4a-86eb-5688b009e4d7 2024-04-24T22:51:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:51:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_33 snap 2024-04-24T22:51:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T22:51:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:51:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_33.export 2024-04-24T22:51:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_33.export 2024-04-24T22:51:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_33.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_33.export 2024-04-24T22:51:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_33.export 2024-04-24T22:51:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:51:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_33.export 2024-04-24T22:51:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:51:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_33.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_33.export 2024-04-24T22:51:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_33.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_33.export 2024-04-24T22:51:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:51:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_34 2024-04-24T22:51:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_34 snap 2024-04-24T22:51:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T22:51:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:51:52.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2024-04-24T22:51:52.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T22:51:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2024-04-24T22:51:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T22:51:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:51:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:51:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:51:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:51:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2024-04-24T22:51:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T22:51:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:51:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:51:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:51:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2024-04-24T22:51:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:51:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:51:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:51:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:51:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:51:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:51:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:51:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:51:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:51:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-04-24T22:51:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:51:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:51:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:51:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:51:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:51:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:51:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:51:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:51:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:51:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:51:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:51:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:51:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:51:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:51:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:51:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:51:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-04-24T22:51:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-24T22:51:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2024-04-24T22:51:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:51:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:51:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:51:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:51:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:51:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:51:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:51:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:51:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:51:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:51:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:51:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:51:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:51:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:51:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:51:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:51:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2024-04-24T22:51:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-24T22:51:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_34 2024-04-24T22:51:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:51:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:51:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:51:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:51:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:51:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:51:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:51:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:51:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:51:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:51:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:51:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:51:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:51:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:51:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:51:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:51:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:51:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_34 2024-04-24T22:51:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-04-24T22:51:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_34 2024-04-24T22:51:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:51:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:51:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:51:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:51:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:51:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:51:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:51:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:51:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:51:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:51:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:51:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:51:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:51:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:51:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:51:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:51:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:51:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_34 2024-04-24T22:51:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:51:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:51:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:51:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:51:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:51:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:51:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:51:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2024-04-24T22:51:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:51:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T22:51:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:51:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:51:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2024-04-24T22:51:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:51:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T22:51:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_34.status 2024-04-24T22:51:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_34.status 2024-04-24T22:51:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:51:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_34.status 2024-04-24T22:51:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_34.status 2024-04-24T22:51:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_34.status 2024-04-24T22:51:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:51:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_34.status 2024-04-24T22:51:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2024-04-24T22:51:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T22:51:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2024-04-24T22:51:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 185 2024-04-24T22:51:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_34.status 2024-04-24T22:51:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T22:51:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_34.status 2024-04-24T22:51:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2024-04-24T22:51:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:51:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=185 2024-04-24T22:51:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:51:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:51:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_34.status 2024-04-24T22:51:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T22:51:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_34.status 2024-04-24T22:51:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-04-24T22:51:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:51:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:51:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:51:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:51:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:51:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 185 = '' 2024-04-24T22:51:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:51:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_34.status 2024-04-24T22:51:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T22:51:55.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_34.status 2024-04-24T22:51:55.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-04-24T22:51:55.251 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:51:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:51:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:51:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:51:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:51:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 185 = '' 2024-04-24T22:51:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:51:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:51:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_34.status 2024-04-24T22:51:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T22:51:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_34.status 2024-04-24T22:51:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-04-24T22:51:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:51:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:51:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=185 2024-04-24T22:51:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 185 = 185 2024-04-24T22:51:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2024-04-24T22:51:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T22:51:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:51:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:51:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:51:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2024-04-24T22:51:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_34': 2024-04-24T22:51:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:51:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:51:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:51:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc544e2ad6 2024-04-24T22:51:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc544e2ad6 2024-04-24T22:51:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:51:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:51:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:51:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:51:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:44:51 2024 2024-04-24T22:51:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:44:51 2024 2024-04-24T22:51:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:44:51 2024 2024-04-24T22:51:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:51:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:51:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:51:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 8081cb7e-e5d5-4fb7-8838-62c6156ec6a4 2024-04-24T22:51:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:51:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_34 snap 2024-04-24T22:51:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T22:51:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:51:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_34.export 2024-04-24T22:51:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_34.export 2024-04-24T22:51:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_34.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_34.export 2024-04-24T22:51:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_34.export 2024-04-24T22:51:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:51:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_34.export 2024-04-24T22:51:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:51:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_34.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_34.export 2024-04-24T22:51:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_34.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_34.export 2024-04-24T22:51:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:51:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:51:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_35 2024-04-24T22:51:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_35 snap 2024-04-24T22:51:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:51:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T22:51:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:51:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:51:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2024-04-24T22:51:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T22:51:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2024-04-24T22:51:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T22:51:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:51:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:51:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:51:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:51:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2024-04-24T22:51:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:51:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:51:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T22:51:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:51:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:51:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:51:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2024-04-24T22:51:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:51:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:51:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:51:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:51:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:51:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:51:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:51:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:51:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:51:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2024-04-24T22:51:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:51:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:51:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:51:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:51:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:51:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:51:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:51:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:51:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:51:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:51:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:51:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:51:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:51:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:51:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:51:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:51:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:51:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2024-04-24T22:51:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-04-24T22:51:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:51:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:51:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:51:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_35 2024-04-24T22:51:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:51:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:51:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:51:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:51:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:51:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:51:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:51:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:51:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:51:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:51:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:51:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:51:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:51:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:51:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:51:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:51:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:51:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:51:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:51:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:51:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:51:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:51:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:51:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:51:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:51:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:51:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:51:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:51:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:51:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:51:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_35 2024-04-24T22:52:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-04-24T22:52:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_35 2024-04-24T22:52:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:52:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:52:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:52:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:52:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:52:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:52:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:52:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:52:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:52:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:52:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:52:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:52:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:52:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:52:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:52:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:52:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_35 2024-04-24T22:52:00.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-04-24T22:52:00.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:00.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_35 2024-04-24T22:52:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:52:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:52:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:52:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:52:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:52:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:52:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:52:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:52:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:52:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:52:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:52:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:00.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:52:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:52:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:52:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:52:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:52:00.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_35 2024-04-24T22:52:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:52:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:52:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:52:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:52:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:52:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:52:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:52:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2024-04-24T22:52:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:52:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T22:52:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:52:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:52:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2024-04-24T22:52:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:52:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T22:52:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_35.status 2024-04-24T22:52:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_35.status 2024-04-24T22:52:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:52:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_35.status 2024-04-24T22:52:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_35.status 2024-04-24T22:52:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_35.status 2024-04-24T22:52:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:52:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_35.status 2024-04-24T22:52:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2024-04-24T22:52:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T22:52:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2024-04-24T22:52:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 187 2024-04-24T22:52:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_35.status 2024-04-24T22:52:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T22:52:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_35.status 2024-04-24T22:52:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2024-04-24T22:52:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:52:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=187 2024-04-24T22:52:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:52:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:52:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_35.status 2024-04-24T22:52:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T22:52:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_35.status 2024-04-24T22:52:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-04-24T22:52:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=102 2024-04-24T22:52:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 187 = 102 2024-04-24T22:52:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:01.145 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:52:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_35.status 2024-04-24T22:52:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T22:52:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_35.status 2024-04-24T22:52:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-04-24T22:52:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=102 2024-04-24T22:52:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 187 = 102 2024-04-24T22:52:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:52:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_35.status 2024-04-24T22:52:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T22:52:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_35.status 2024-04-24T22:52:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-04-24T22:52:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:02.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:52:02.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:52:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:52:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:52:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:52:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 187 = '' 2024-04-24T22:52:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T22:52:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_35.status 2024-04-24T22:52:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T22:52:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_35.status 2024-04-24T22:52:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-04-24T22:52:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=187 2024-04-24T22:52:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 187 = 187 2024-04-24T22:52:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2024-04-24T22:52:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T22:52:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:52:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:52:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:52:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2024-04-24T22:52:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_35': 2024-04-24T22:52:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:52:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:52:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:52:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc114c0a93 2024-04-24T22:52:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc114c0a93 2024-04-24T22:52:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:52:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:52:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:52:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:52:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:44:57 2024 2024-04-24T22:52:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:44:57 2024 2024-04-24T22:52:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:44:57 2024 2024-04-24T22:52:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:52:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:52:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:52:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 77b74900-6e9c-4b05-b994-e2ce65e571fa 2024-04-24T22:52:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:52:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_35 snap 2024-04-24T22:52:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T22:52:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:52:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_35.export 2024-04-24T22:52:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_35.export 2024-04-24T22:52:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_35.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_35.export 2024-04-24T22:52:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_35.export 2024-04-24T22:52:05.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:52:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_35.export 2024-04-24T22:52:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:52:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_35.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_35.export 2024-04-24T22:52:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_35.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_35.export 2024-04-24T22:52:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:52:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_36 2024-04-24T22:52:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_36 snap 2024-04-24T22:52:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T22:52:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:52:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2024-04-24T22:52:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T22:52:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2024-04-24T22:52:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T22:52:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:52:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:52:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:52:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:52:07.671 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-24T22:52:07.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-24T22:52:07.672 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-04-24T22:52:07.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T22:52:07.671+0000 7f83b46e8700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 67620) UID: 1000 2024-04-24T22:52:07.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:52:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2024-04-24T22:52:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T22:52:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:52:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:52:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:52:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2024-04-24T22:52:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:52:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:52:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:52:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:52:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:52:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:52:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:52:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:52:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:52:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-04-24T22:52:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:52:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:52:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:52:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:52:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:52:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:52:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:52:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:52:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:52:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:52:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:52:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:52:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:52:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:52:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:52:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:52:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-04-24T22:52:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T22:52:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-04-24T22:52:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:52:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:52:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:52:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:52:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:52:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:52:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:52:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:52:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:52:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:52:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:52:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:52:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:52:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:52:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:52:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:52:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-04-24T22:52:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-04-24T22:52:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2024-04-24T22:52:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:52:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:52:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:52:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:52:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:52:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:52:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:52:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:52:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:52:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:52:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:52:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:52:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:52:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:52:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:52:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:52:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2024-04-24T22:52:08.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-04-24T22:52:08.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:08.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_36 2024-04-24T22:52:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:52:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:52:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:52:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:52:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:52:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:52:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:52:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:52:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:52:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:52:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:52:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:52:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:52:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:52:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:52:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:52:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_36 2024-04-24T22:52:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:52:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:52:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:52:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:52:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:52:08.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:52:08.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:52:08.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:08.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2024-04-24T22:52:08.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:52:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T22:52:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:52:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:52:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2024-04-24T22:52:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:52:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T22:52:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_36.status 2024-04-24T22:52:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_36.status 2024-04-24T22:52:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:52:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_36.status 2024-04-24T22:52:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_36.status 2024-04-24T22:52:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_36.status 2024-04-24T22:52:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:52:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_36.status 2024-04-24T22:52:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2024-04-24T22:52:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T22:52:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2024-04-24T22:52:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 189 2024-04-24T22:52:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_36.status 2024-04-24T22:52:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T22:52:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_36.status 2024-04-24T22:52:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2024-04-24T22:52:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:52:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=189 2024-04-24T22:52:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:52:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:52:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_36.status 2024-04-24T22:52:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T22:52:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_36.status 2024-04-24T22:52:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-04-24T22:52:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=103 2024-04-24T22:52:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 189 = 103 2024-04-24T22:52:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:52:10.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_36.status 2024-04-24T22:52:10.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:10.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:10.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T22:52:10.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_36.status 2024-04-24T22:52:10.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-04-24T22:52:10.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:52:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:52:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:52:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:52:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:52:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 189 = '' 2024-04-24T22:52:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:52:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_36.status 2024-04-24T22:52:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T22:52:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_36.status 2024-04-24T22:52:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-04-24T22:52:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=189 2024-04-24T22:52:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 189 = 189 2024-04-24T22:52:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2024-04-24T22:52:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T22:52:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:52:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:52:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:52:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2024-04-24T22:52:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_36': 2024-04-24T22:52:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:52:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:52:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:52:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc9ff72ff6 2024-04-24T22:52:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc9ff72ff6 2024-04-24T22:52:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:52:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:52:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:52:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:52:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:45:09 2024 2024-04-24T22:52:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:45:09 2024 2024-04-24T22:52:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:45:09 2024 2024-04-24T22:52:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:52:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:52:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:52:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: d616d270-b42d-464e-90d2-860a74167c56 2024-04-24T22:52:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:52:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_36 snap 2024-04-24T22:52:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T22:52:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:52:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_36.export 2024-04-24T22:52:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_36.export 2024-04-24T22:52:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_36.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_36.export 2024-04-24T22:52:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_36.export 2024-04-24T22:52:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:52:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_36.export 2024-04-24T22:52:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:52:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_36.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_36.export 2024-04-24T22:52:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_36.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_36.export 2024-04-24T22:52:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:52:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_37 2024-04-24T22:52:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_37 snap 2024-04-24T22:52:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T22:52:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:52:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2024-04-24T22:52:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T22:52:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2024-04-24T22:52:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T22:52:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:52:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:52:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:52:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:52:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2024-04-24T22:52:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T22:52:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:52:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:52:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:52:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2024-04-24T22:52:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:52:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:52:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:52:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:52:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:52:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:52:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:52:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:52:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:52:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2024-04-24T22:52:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:52:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:52:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:52:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:52:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:52:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:52:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:52:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:52:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:52:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:52:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:52:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:52:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:52:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:52:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:52:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:52:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2024-04-24T22:52:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T22:52:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2024-04-24T22:52:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:52:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:52:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:52:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:52:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:52:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:52:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:52:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:52:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:52:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:52:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:52:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:52:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:52:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:52:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:52:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:52:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2024-04-24T22:52:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-24T22:52:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_37 2024-04-24T22:52:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:52:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:52:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:52:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:52:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:52:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:52:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:52:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:52:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:52:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:52:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:52:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:52:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:52:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:52:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:52:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:52:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_37 2024-04-24T22:52:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-04-24T22:52:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_37 2024-04-24T22:52:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:52:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:52:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:52:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:52:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:52:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:52:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:52:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:52:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:52:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:52:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:52:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:52:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:52:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:52:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:52:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:52:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_37 2024-04-24T22:52:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:52:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:52:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:52:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:52:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:52:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:52:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:52:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2024-04-24T22:52:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:52:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T22:52:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:52:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:52:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2024-04-24T22:52:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:52:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T22:52:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_37.status 2024-04-24T22:52:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_37.status 2024-04-24T22:52:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:52:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_37.status 2024-04-24T22:52:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_37.status 2024-04-24T22:52:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_37.status 2024-04-24T22:52:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:52:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_37.status 2024-04-24T22:52:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2024-04-24T22:52:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T22:52:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2024-04-24T22:52:16.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 191 2024-04-24T22:52:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_37.status 2024-04-24T22:52:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:16.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T22:52:16.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_37.status 2024-04-24T22:52:16.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2024-04-24T22:52:16.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:52:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=191 2024-04-24T22:52:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:52:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:52:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_37.status 2024-04-24T22:52:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T22:52:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_37.status 2024-04-24T22:52:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-04-24T22:52:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=104 2024-04-24T22:52:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 191 = 104 2024-04-24T22:52:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:52:16.673 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-24T22:52:16.673 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:52:16.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:52:16.673 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-24T22:52:16.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-24T22:52:16.674 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-24T22:52:16.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-24T22:52:16.678 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-04-24T22:52:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_37.status 2024-04-24T22:52:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T22:52:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_37.status 2024-04-24T22:52:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-04-24T22:52:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:52:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:52:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:52:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:52:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:52:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 191 = '' 2024-04-24T22:52:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:52:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_37.status 2024-04-24T22:52:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T22:52:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_37.status 2024-04-24T22:52:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-04-24T22:52:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=191 2024-04-24T22:52:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 191 = 191 2024-04-24T22:52:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2024-04-24T22:52:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T22:52:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:52:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:52:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:52:18.540 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4994319764860464) > weight (0.25) 2024-04-24T22:52:18.540 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8556168057005141) > weight (0.25) 2024-04-24T22:52:18.540 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5441502245475431) > weight (0.25) 2024-04-24T22:52:18.540 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6783926739154682) > weight (0.25) 2024-04-24T22:52:18.540 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2024-04-24T22:52:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2024-04-24T22:52:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_37': 2024-04-24T22:52:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:52:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:52:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:52:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc643485b7 2024-04-24T22:52:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc643485b7 2024-04-24T22:52:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:52:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:52:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:52:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:52:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:45:17 2024 2024-04-24T22:52:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:45:17 2024 2024-04-24T22:52:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:45:17 2024 2024-04-24T22:52:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:52:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:52:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:52:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: a3bbbaa2-a609-401b-8231-3adcfdc73de8 2024-04-24T22:52:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:52:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_37 snap 2024-04-24T22:52:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T22:52:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:52:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_37.export 2024-04-24T22:52:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_37.export 2024-04-24T22:52:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_37.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_37.export 2024-04-24T22:52:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_37.export 2024-04-24T22:52:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:52:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_37.export 2024-04-24T22:52:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:52:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_37.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_37.export 2024-04-24T22:52:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_37.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_37.export 2024-04-24T22:52:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:52:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_38 2024-04-24T22:52:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_38 snap 2024-04-24T22:52:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T22:52:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:52:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2024-04-24T22:52:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T22:52:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2024-04-24T22:52:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T22:52:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:52:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:52:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:52:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:52:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2024-04-24T22:52:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T22:52:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:52:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:52:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:52:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2024-04-24T22:52:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:52:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:52:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:52:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:52:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:52:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:52:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:52:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:52:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:52:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2024-04-24T22:52:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:52:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:52:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:52:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:52:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:52:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:52:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:52:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:52:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:52:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:52:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:52:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:52:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:52:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:52:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:52:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:52:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2024-04-24T22:52:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-04-24T22:52:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2024-04-24T22:52:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:52:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:52:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:52:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:52:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:52:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:52:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:52:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:52:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:52:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:52:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:52:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:52:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:52:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:52:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:52:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:52:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:21.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2024-04-24T22:52:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-04-24T22:52:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2024-04-24T22:52:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:52:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:52:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:52:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:52:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:52:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:52:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:52:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:52:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:52:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:52:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:52:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:52:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:52:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:52:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:52:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:52:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2024-04-24T22:52:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-04-24T22:52:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_38 2024-04-24T22:52:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:52:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:52:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:52:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:52:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:52:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:52:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:52:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:52:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:52:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:52:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:52:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:52:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:52:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:52:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:52:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:52:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_38 2024-04-24T22:52:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:52:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:52:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:52:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:52:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:52:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:52:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:52:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2024-04-24T22:52:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:52:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T22:52:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:52:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:52:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2024-04-24T22:52:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:52:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T22:52:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_38.status 2024-04-24T22:52:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_38.status 2024-04-24T22:52:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:52:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_38.status 2024-04-24T22:52:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_38.status 2024-04-24T22:52:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_38.status 2024-04-24T22:52:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:52:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_38.status 2024-04-24T22:52:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2024-04-24T22:52:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T22:52:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2024-04-24T22:52:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 193 2024-04-24T22:52:22.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_38.status 2024-04-24T22:52:22.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T22:52:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_38.status 2024-04-24T22:52:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2024-04-24T22:52:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:22.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:52:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=193 2024-04-24T22:52:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:52:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:52:22.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_38.status 2024-04-24T22:52:22.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:22.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:22.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T22:52:22.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_38.status 2024-04-24T22:52:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-04-24T22:52:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=105 2024-04-24T22:52:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 193 = 105 2024-04-24T22:52:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:52:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_38.status 2024-04-24T22:52:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T22:52:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_38.status 2024-04-24T22:52:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-04-24T22:52:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:52:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:52:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:52:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:52:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:52:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 193 = '' 2024-04-24T22:52:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:52:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_38.status 2024-04-24T22:52:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T22:52:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_38.status 2024-04-24T22:52:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-04-24T22:52:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=193 2024-04-24T22:52:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 193 = 193 2024-04-24T22:52:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2024-04-24T22:52:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T22:52:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:52:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:52:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:52:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2024-04-24T22:52:25.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_38': 2024-04-24T22:52:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:52:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:52:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:52:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc15bc5a6f 2024-04-24T22:52:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc15bc5a6f 2024-04-24T22:52:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:52:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:52:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:52:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:52:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:45:27 2024 2024-04-24T22:52:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:45:27 2024 2024-04-24T22:52:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:45:27 2024 2024-04-24T22:52:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:52:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:52:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:52:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 0b0a4d2e-042e-4de3-97db-78d53010573b 2024-04-24T22:52:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:52:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_38 snap 2024-04-24T22:52:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T22:52:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:52:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_38.export 2024-04-24T22:52:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_38.export 2024-04-24T22:52:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_38.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_38.export 2024-04-24T22:52:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_38.export 2024-04-24T22:52:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:52:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_38.export 2024-04-24T22:52:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:52:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_38.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_38.export 2024-04-24T22:52:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_38.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_38.export 2024-04-24T22:52:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:52:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_39 2024-04-24T22:52:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_39 snap 2024-04-24T22:52:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T22:52:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:52:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2024-04-24T22:52:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T22:52:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2024-04-24T22:52:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T22:52:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:52:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:52:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:52:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:52:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2024-04-24T22:52:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T22:52:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:52:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:52:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:52:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2024-04-24T22:52:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:52:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:52:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:52:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:52:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:52:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:52:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:52:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:52:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:52:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-04-24T22:52:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:52:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:52:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:52:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:52:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:52:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:52:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:52:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:52:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:52:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:52:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:52:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:52:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:52:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:52:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:52:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:52:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-04-24T22:52:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-04-24T22:52:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2024-04-24T22:52:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:52:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:52:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:52:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:52:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:52:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:52:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:52:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:52:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:52:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:52:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:52:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:52:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:52:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:52:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:52:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:52:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2024-04-24T22:52:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-04-24T22:52:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_39 2024-04-24T22:52:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:52:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:52:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:52:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:52:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:52:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:52:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:52:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:52:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:52:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:52:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:52:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:52:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:52:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:52:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:52:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:52:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_39 2024-04-24T22:52:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-04-24T22:52:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_39 2024-04-24T22:52:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:52:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:52:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:52:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:52:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:52:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:52:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:52:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:52:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:52:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:52:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:52:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:52:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:52:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:52:28.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:52:28.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:52:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_39 2024-04-24T22:52:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:52:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:52:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:52:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:52:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:52:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:52:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:52:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2024-04-24T22:52:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:52:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T22:52:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:52:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:52:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2024-04-24T22:52:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:52:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T22:52:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_39.status 2024-04-24T22:52:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_39.status 2024-04-24T22:52:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:52:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_39.status 2024-04-24T22:52:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_39.status 2024-04-24T22:52:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_39.status 2024-04-24T22:52:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:52:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_39.status 2024-04-24T22:52:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2024-04-24T22:52:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T22:52:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2024-04-24T22:52:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 195 2024-04-24T22:52:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_39.status 2024-04-24T22:52:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T22:52:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_39.status 2024-04-24T22:52:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2024-04-24T22:52:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:52:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=195 2024-04-24T22:52:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:52:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:52:29.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_39.status 2024-04-24T22:52:29.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:29.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:29.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T22:52:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_39.status 2024-04-24T22:52:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-04-24T22:52:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:29.098 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=106 2024-04-24T22:52:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 195 = 106 2024-04-24T22:52:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:52:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_39.status 2024-04-24T22:52:29.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:29.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:29.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T22:52:29.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_39.status 2024-04-24T22:52:29.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-04-24T22:52:29.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:52:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:29.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:52:29.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:52:29.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:52:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:52:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 195 = '' 2024-04-24T22:52:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:52:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_39.status 2024-04-24T22:52:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T22:52:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_39.status 2024-04-24T22:52:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-04-24T22:52:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:30.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:52:30.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:52:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:52:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:52:30.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:52:30.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 195 = '' 2024-04-24T22:52:30.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:30.414 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T22:52:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_39.status 2024-04-24T22:52:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T22:52:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_39.status 2024-04-24T22:52:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-04-24T22:52:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:32.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=195 2024-04-24T22:52:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 195 = 195 2024-04-24T22:52:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2024-04-24T22:52:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T22:52:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:52:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:52:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:52:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2024-04-24T22:52:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_39': 2024-04-24T22:52:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:52:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:52:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:52:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc23c28715 2024-04-24T22:52:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc23c28715 2024-04-24T22:52:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:52:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:52:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:52:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:52:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:45:33 2024 2024-04-24T22:52:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:45:33 2024 2024-04-24T22:52:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:45:33 2024 2024-04-24T22:52:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:52:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:52:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:52:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 74beb076-4d18-4365-b1a8-d0b50a2536f5 2024-04-24T22:52:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:52:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_39 snap 2024-04-24T22:52:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T22:52:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:52:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_39.export 2024-04-24T22:52:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_39.export 2024-04-24T22:52:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_39.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_39.export 2024-04-24T22:52:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_39.export 2024-04-24T22:52:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:52:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_39.export 2024-04-24T22:52:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:52:33.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_39.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_39.export 2024-04-24T22:52:33.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_39.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_39.export 2024-04-24T22:52:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:52:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_40 2024-04-24T22:52:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_40 snap 2024-04-24T22:52:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:33.807 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T22:52:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:52:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2024-04-24T22:52:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T22:52:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2024-04-24T22:52:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T22:52:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:52:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:52:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:52:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:52:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2024-04-24T22:52:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T22:52:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:52:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:52:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:52:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2024-04-24T22:52:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:52:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:52:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:52:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:52:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:52:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:52:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:52:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:52:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:52:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2024-04-24T22:52:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:52:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:52:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:52:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:52:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:52:36.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:52:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:52:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:52:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:52:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:52:36.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:52:36.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:36.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:36.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:36.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:52:36.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:52:36.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:36.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:36.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:52:36.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:52:36.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:52:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2024-04-24T22:52:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-04-24T22:52:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:36.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2024-04-24T22:52:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:52:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:52:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:52:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:52:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:52:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:52:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:52:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:52:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:52:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:52:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:52:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:52:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:52:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:52:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:52:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:52:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2024-04-24T22:52:36.514 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:52:36.513+0000 7f0f4ef7b700 -1 reset not still connected to 0x561388b12f70 2024-04-24T22:52:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-04-24T22:52:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_40 2024-04-24T22:52:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:52:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:52:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:52:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:52:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:52:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:52:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:52:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:52:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:52:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:52:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:52:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:52:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:52:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:52:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:52:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:52:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_40 2024-04-24T22:52:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-04-24T22:52:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_40 2024-04-24T22:52:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:52:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:52:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:52:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:52:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:52:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:52:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:52:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:52:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:52:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:52:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:52:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:52:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:52:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:52:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:52:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:52:36.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_40 2024-04-24T22:52:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:52:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:52:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:52:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:52:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:52:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:52:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:52:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2024-04-24T22:52:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:52:36.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T22:52:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:52:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:52:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2024-04-24T22:52:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:52:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T22:52:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_40.status 2024-04-24T22:52:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_40.status 2024-04-24T22:52:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:52:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_40.status 2024-04-24T22:52:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_40.status 2024-04-24T22:52:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_40.status 2024-04-24T22:52:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:52:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_40.status 2024-04-24T22:52:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2024-04-24T22:52:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T22:52:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2024-04-24T22:52:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 197 2024-04-24T22:52:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_40.status 2024-04-24T22:52:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T22:52:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_40.status 2024-04-24T22:52:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2024-04-24T22:52:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:52:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=197 2024-04-24T22:52:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:52:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:52:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_40.status 2024-04-24T22:52:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T22:52:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_40.status 2024-04-24T22:52:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-04-24T22:52:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:52:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:37.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:52:37.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:52:37.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:52:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:52:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 197 = '' 2024-04-24T22:52:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:52:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_40.status 2024-04-24T22:52:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T22:52:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_40.status 2024-04-24T22:52:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-04-24T22:52:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:52:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:52:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:52:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:52:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:52:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 197 = '' 2024-04-24T22:52:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:52:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_40.status 2024-04-24T22:52:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T22:52:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_40.status 2024-04-24T22:52:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-04-24T22:52:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=197 2024-04-24T22:52:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 197 = 197 2024-04-24T22:52:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2024-04-24T22:52:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T22:52:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:52:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:52:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:52:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2024-04-24T22:52:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_40': 2024-04-24T22:52:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:52:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:52:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:52:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dcaf63fd4b 2024-04-24T22:52:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dcaf63fd4b 2024-04-24T22:52:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:52:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:52:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:52:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:52:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:45:40 2024 2024-04-24T22:52:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:45:40 2024 2024-04-24T22:52:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:45:40 2024 2024-04-24T22:52:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:52:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:52:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:52:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: b9d9d1d6-592b-471e-8736-169dcce0a795 2024-04-24T22:52:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:52:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_40 snap 2024-04-24T22:52:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T22:52:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:52:39.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_40.export 2024-04-24T22:52:39.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_40.export 2024-04-24T22:52:39.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_40.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_40.export 2024-04-24T22:52:39.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_40.export 2024-04-24T22:52:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:52:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_40.export 2024-04-24T22:52:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:52:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_40.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_40.export 2024-04-24T22:52:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_40.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_40.export 2024-04-24T22:52:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:52:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_41 2024-04-24T22:52:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_41 snap 2024-04-24T22:52:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T22:52:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:52:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2024-04-24T22:52:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T22:52:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2024-04-24T22:52:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T22:52:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:52:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:52:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:52:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:52:43.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2024-04-24T22:52:43.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:43.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T22:52:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:52:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:52:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:52:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2024-04-24T22:52:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:52:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:52:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:52:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:52:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:52:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:52:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:52:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:52:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:52:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-04-24T22:52:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:52:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:52:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:52:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:52:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:52:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:52:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:52:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:52:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:52:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:52:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:52:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:52:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:52:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:52:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:52:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:52:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-04-24T22:52:43.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-04-24T22:52:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2024-04-24T22:52:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:52:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:52:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:52:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:52:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:52:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:52:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:52:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:52:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:52:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:52:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:52:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:52:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:52:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:52:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:52:43.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:52:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2024-04-24T22:52:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-04-24T22:52:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2024-04-24T22:52:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:52:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:52:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:52:43.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:52:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:52:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:52:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:52:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:52:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:52:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:52:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:52:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:52:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:52:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:52:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:52:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:52:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2024-04-24T22:52:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-04-24T22:52:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_41 2024-04-24T22:52:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:52:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:52:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:52:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:52:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:52:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:52:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:52:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:52:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:52:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:52:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:52:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:52:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:52:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:52:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:52:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:52:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_41 2024-04-24T22:52:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:52:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:52:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:52:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:52:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:52:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:52:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:52:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2024-04-24T22:52:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:52:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T22:52:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:52:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:52:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2024-04-24T22:52:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:52:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T22:52:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_41.status 2024-04-24T22:52:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_41.status 2024-04-24T22:52:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:52:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_41.status 2024-04-24T22:52:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_41.status 2024-04-24T22:52:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_41.status 2024-04-24T22:52:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:52:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_41.status 2024-04-24T22:52:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2024-04-24T22:52:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T22:52:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2024-04-24T22:52:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 199 2024-04-24T22:52:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_41.status 2024-04-24T22:52:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T22:52:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_41.status 2024-04-24T22:52:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2024-04-24T22:52:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:52:44.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=199 2024-04-24T22:52:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:52:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:52:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_41.status 2024-04-24T22:52:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T22:52:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_41.status 2024-04-24T22:52:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-04-24T22:52:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:52:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:44.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:52:44.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:52:44.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:52:44.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:52:44.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 199 = '' 2024-04-24T22:52:44.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:52:45.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_41.status 2024-04-24T22:52:45.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:45.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:45.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T22:52:45.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_41.status 2024-04-24T22:52:45.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-04-24T22:52:45.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:52:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:52:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:52:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:52:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:52:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 199 = '' 2024-04-24T22:52:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:52:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_41.status 2024-04-24T22:52:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T22:52:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_41.status 2024-04-24T22:52:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-04-24T22:52:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=199 2024-04-24T22:52:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 199 = 199 2024-04-24T22:52:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2024-04-24T22:52:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T22:52:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:52:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:52:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:52:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2024-04-24T22:52:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_41': 2024-04-24T22:52:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:52:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:52:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:52:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc81b898b1 2024-04-24T22:52:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc81b898b1 2024-04-24T22:52:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:52:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:52:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:52:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:52:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:45:54 2024 2024-04-24T22:52:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:45:54 2024 2024-04-24T22:52:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:45:54 2024 2024-04-24T22:52:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:52:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:52:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:52:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: f62d1554-3ea2-4d48-bd05-d49435911c52 2024-04-24T22:52:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:52:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_41 snap 2024-04-24T22:52:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T22:52:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:52:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_41.export 2024-04-24T22:52:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_41.export 2024-04-24T22:52:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_41.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_41.export 2024-04-24T22:52:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_41.export 2024-04-24T22:52:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:52:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_41.export 2024-04-24T22:52:47.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:52:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_41.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_41.export 2024-04-24T22:52:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_41.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_41.export 2024-04-24T22:52:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:52:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_42 2024-04-24T22:52:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_42 snap 2024-04-24T22:52:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T22:52:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:52:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2024-04-24T22:52:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T22:52:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2024-04-24T22:52:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T22:52:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:52:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:52:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:52:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:52:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2024-04-24T22:52:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T22:52:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:52:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:52:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:52:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2024-04-24T22:52:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:52:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:52:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:52:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:52:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:52:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:52:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:52:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:52:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:52:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2024-04-24T22:52:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:52:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:52:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:52:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:52:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:52:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:52:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:52:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:52:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:52:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:52:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:52:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:52:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:52:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:52:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:52:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:52:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2024-04-24T22:52:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-04-24T22:52:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2024-04-24T22:52:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:52:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:52:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:52:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:52:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:52:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:52:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:52:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:52:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:52:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:52:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:52:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:52:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:52:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:52:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:52:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:52:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2024-04-24T22:52:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-04-24T22:52:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_42 2024-04-24T22:52:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:52:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:52:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:52:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:52:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:52:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:52:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:52:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:52:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:52:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:52:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:52:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:52:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:52:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:52:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:52:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:52:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_42 2024-04-24T22:52:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-04-24T22:52:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_42 2024-04-24T22:52:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:52:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:52:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:52:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:52:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:52:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:52:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:52:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:52:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:52:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:52:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:52:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:52:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:52:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:52:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:52:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:52:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_42 2024-04-24T22:52:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:52:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:52:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:52:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:52:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:52:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:52:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:52:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2024-04-24T22:52:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:52:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T22:52:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:52:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:52:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2024-04-24T22:52:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:52:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T22:52:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_42.status 2024-04-24T22:52:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_42.status 2024-04-24T22:52:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:52:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_42.status 2024-04-24T22:52:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_42.status 2024-04-24T22:52:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_42.status 2024-04-24T22:52:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:52:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_42.status 2024-04-24T22:52:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2024-04-24T22:52:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T22:52:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2024-04-24T22:52:50.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 201 2024-04-24T22:52:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_42.status 2024-04-24T22:52:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T22:52:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_42.status 2024-04-24T22:52:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2024-04-24T22:52:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:52:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=201 2024-04-24T22:52:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:52:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:50.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:52:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_42.status 2024-04-24T22:52:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T22:52:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_42.status 2024-04-24T22:52:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-04-24T22:52:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=109 2024-04-24T22:52:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 201 = 109 2024-04-24T22:52:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:52:51.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_42.status 2024-04-24T22:52:51.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:51.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:51.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T22:52:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_42.status 2024-04-24T22:52:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-04-24T22:52:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=109 2024-04-24T22:52:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 201 = 109 2024-04-24T22:52:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:52:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_42.status 2024-04-24T22:52:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T22:52:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_42.status 2024-04-24T22:52:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-04-24T22:52:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:52:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:52:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:52:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:52:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:52:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 201 = '' 2024-04-24T22:52:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T22:52:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_42.status 2024-04-24T22:52:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:53.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T22:52:53.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_42.status 2024-04-24T22:52:53.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-04-24T22:52:53.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=201 2024-04-24T22:52:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 201 = 201 2024-04-24T22:52:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2024-04-24T22:52:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T22:52:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:53.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:52:53.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:52:53.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:52:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2024-04-24T22:52:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_42': 2024-04-24T22:52:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:52:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:52:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:52:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc41075e 2024-04-24T22:52:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc41075e 2024-04-24T22:52:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:52:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:52:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:52:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:52:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:46:04 2024 2024-04-24T22:52:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:46:04 2024 2024-04-24T22:52:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:46:04 2024 2024-04-24T22:52:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:52:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:52:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:52:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: abffceee-6d69-43b8-8792-bb1ec5129a1f 2024-04-24T22:52:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:52:54.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_42 snap 2024-04-24T22:52:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T22:52:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:52:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_42.export 2024-04-24T22:52:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_42.export 2024-04-24T22:52:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_42.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_42.export 2024-04-24T22:52:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_42.export 2024-04-24T22:52:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:52:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_42.export 2024-04-24T22:52:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:52:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_42.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_42.export 2024-04-24T22:52:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_42.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_42.export 2024-04-24T22:52:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:52:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_43 2024-04-24T22:52:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_43 snap 2024-04-24T22:52:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T22:52:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:52:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:52:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2024-04-24T22:52:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T22:52:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2024-04-24T22:52:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T22:52:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:52:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:52:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:52:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:52:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2024-04-24T22:52:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T22:52:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:52:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:52:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:52:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2024-04-24T22:52:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:52:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:52:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:52:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:52:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:52:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:52:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:52:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:52:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:52:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2024-04-24T22:52:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:52:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:52:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:52:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:52:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:52:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:52:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:52:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:52:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:52:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:52:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:52:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:52:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:52:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:52:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:52:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:52:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:52:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2024-04-24T22:52:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-04-24T22:52:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2024-04-24T22:52:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:52:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:52:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:52:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:52:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:52:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:52:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:52:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:52:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:52:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:52:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:52:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:52:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:52:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:52:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:52:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:52:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:52:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2024-04-24T22:52:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-04-24T22:52:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_43 2024-04-24T22:52:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:52:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:52:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:52:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:52:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:52:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:52:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:52:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:52:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:52:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:52:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:52:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:52:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:52:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:52:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:52:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:52:57.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:57.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:57.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:52:57.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_43 2024-04-24T22:52:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-04-24T22:52:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:52:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:52:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:52:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_43 2024-04-24T22:52:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:52:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:52:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:52:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:52:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:52:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:52:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:52:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:52:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:52:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:52:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:52:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:52:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:52:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:52:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:52:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:52:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:52:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:52:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:52:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:52:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:52:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:52:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:52:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:52:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:52:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:52:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:52:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:52:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:52:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:52:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_43 2024-04-24T22:52:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:52:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:52:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:52:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:52:58.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:52:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:52:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:52:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:52:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2024-04-24T22:52:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:52:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T22:52:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:52:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:52:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2024-04-24T22:52:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:52:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T22:52:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_43.status 2024-04-24T22:52:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_43.status 2024-04-24T22:52:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:52:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_43.status 2024-04-24T22:52:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_43.status 2024-04-24T22:52:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_43.status 2024-04-24T22:52:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:52:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_43.status 2024-04-24T22:52:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2024-04-24T22:52:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T22:52:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2024-04-24T22:52:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 203 2024-04-24T22:52:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_43.status 2024-04-24T22:52:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:52:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T22:52:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_43.status 2024-04-24T22:52:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2024-04-24T22:52:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:58.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:52:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=203 2024-04-24T22:52:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:52:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:52:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_43.status 2024-04-24T22:52:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T22:52:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_43.status 2024-04-24T22:52:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-04-24T22:52:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=110 2024-04-24T22:52:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 203 = 110 2024-04-24T22:52:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:52:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_43.status 2024-04-24T22:52:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:52:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:52:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T22:52:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_43.status 2024-04-24T22:52:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-04-24T22:52:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:52:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:52:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=110 2024-04-24T22:52:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 203 = 110 2024-04-24T22:52:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:52:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:53:00.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_43.status 2024-04-24T22:53:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T22:53:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_43.status 2024-04-24T22:53:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-04-24T22:53:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:53:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:00.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:53:00.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:53:00.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:53:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:53:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 203 = '' 2024-04-24T22:53:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T22:53:00.557 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:53:00.552+0000 7f0f4ef7b700 -1 reset not still connected to 0x56139637cf70 2024-04-24T22:53:00.558 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:53:00.554+0000 7f0f4ef7b700 -1 reset not still connected to 0x56139adb3790 2024-04-24T22:53:00.559 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:53:00.558+0000 7f0f4ef7b700 -1 reset not still connected to 0x56139b3fcf70 2024-04-24T22:53:00.559 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:53:00.558+0000 7f0f4ef7b700 -1 reset not still connected to 0x56139b3fd930 2024-04-24T22:53:00.559 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:53:00.558+0000 7f0f4ef7b700 -1 reset not still connected to 0x56139be86820 2024-04-24T22:53:00.559 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:53:00.558+0000 7f0f4ef7b700 -1 reset not still connected to 0x5613aa4f9d40 2024-04-24T22:53:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_43.status 2024-04-24T22:53:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T22:53:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_43.status 2024-04-24T22:53:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-04-24T22:53:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:01.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=203 2024-04-24T22:53:01.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 203 = 203 2024-04-24T22:53:01.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:01.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2024-04-24T22:53:01.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:01.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:01.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T22:53:01.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:53:01.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:53:01.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:53:01.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:53:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2024-04-24T22:53:02.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_43': 2024-04-24T22:53:02.946 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:53:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:53:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:53:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc2a023088 2024-04-24T22:53:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc2a023088 2024-04-24T22:53:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:53:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:53:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:53:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:53:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:46:15 2024 2024-04-24T22:53:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:46:15 2024 2024-04-24T22:53:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:46:15 2024 2024-04-24T22:53:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:53:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:53:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:53:02.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 50cbbefd-55ee-46b7-9b9e-c53db4b8ed79 2024-04-24T22:53:02.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:53:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_43 snap 2024-04-24T22:53:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T22:53:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:53:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:53:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_43.export 2024-04-24T22:53:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_43.export 2024-04-24T22:53:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_43.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_43.export 2024-04-24T22:53:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_43.export 2024-04-24T22:53:03.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:53:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_43.export 2024-04-24T22:53:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:53:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_43.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_43.export 2024-04-24T22:53:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_43.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_43.export 2024-04-24T22:53:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:53:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_44 2024-04-24T22:53:03.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_44 snap 2024-04-24T22:53:03.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:03.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:03.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T22:53:03.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:53:03.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:53:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2024-04-24T22:53:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T22:53:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2024-04-24T22:53:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T22:53:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:53:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:53:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:53:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:53:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2024-04-24T22:53:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T22:53:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:53:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:53:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:53:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2024-04-24T22:53:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:53:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:53:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:53:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:53:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:53:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:53:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:53:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:53:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:53:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2024-04-24T22:53:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:53:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:53:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:53:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:53:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:53:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:53:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:53:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:53:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:53:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:53:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:53:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:53:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:53:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:53:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:53:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:53:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:53:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:53:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:53:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2024-04-24T22:53:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-04-24T22:53:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:53:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:53:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2024-04-24T22:53:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:53:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:53:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:53:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:53:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:53:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:53:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:53:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:53:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:53:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:53:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:53:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:53:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:53:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:53:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:53:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:53:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:53:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:53:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:53:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2024-04-24T22:53:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-04-24T22:53:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:53:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:53:06.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_44 2024-04-24T22:53:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:53:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:53:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:53:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:53:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:53:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:53:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:53:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:53:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:53:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:53:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:53:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:06.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:53:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:53:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:53:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:53:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:53:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:53:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:53:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:53:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_44 2024-04-24T22:53:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-04-24T22:53:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:53:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:53:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_44 2024-04-24T22:53:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:53:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:53:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:53:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:53:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:53:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:53:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:53:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:53:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:53:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:53:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:53:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:53:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:53:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:53:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:53:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:53:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:53:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:53:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:53:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_44 2024-04-24T22:53:06.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:53:06.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:53:06.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:53:06.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:53:06.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:53:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:53:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:53:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2024-04-24T22:53:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:53:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T22:53:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:53:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:53:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2024-04-24T22:53:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:53:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T22:53:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_44.status 2024-04-24T22:53:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_44.status 2024-04-24T22:53:06.837 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:53:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_44.status 2024-04-24T22:53:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_44.status 2024-04-24T22:53:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_44.status 2024-04-24T22:53:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:53:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_44.status 2024-04-24T22:53:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2024-04-24T22:53:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T22:53:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2024-04-24T22:53:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 205 2024-04-24T22:53:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_44.status 2024-04-24T22:53:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T22:53:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_44.status 2024-04-24T22:53:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2024-04-24T22:53:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:53:07.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=205 2024-04-24T22:53:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:53:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:53:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_44.status 2024-04-24T22:53:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T22:53:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_44.status 2024-04-24T22:53:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-04-24T22:53:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:53:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:53:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:53:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:53:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:53:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 205 = '' 2024-04-24T22:53:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:53:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_44.status 2024-04-24T22:53:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T22:53:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_44.status 2024-04-24T22:53:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-04-24T22:53:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:53:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:53:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:53:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:53:08.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:53:08.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 205 = '' 2024-04-24T22:53:08.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:08.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:53:08.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_44.status 2024-04-24T22:53:08.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:08.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:08.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T22:53:08.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_44.status 2024-04-24T22:53:08.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-04-24T22:53:08.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=205 2024-04-24T22:53:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 205 = 205 2024-04-24T22:53:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2024-04-24T22:53:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T22:53:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:53:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:53:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:53:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:53:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2024-04-24T22:53:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_44': 2024-04-24T22:53:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:53:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:53:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:53:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc2f10ae01 2024-04-24T22:53:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc2f10ae01 2024-04-24T22:53:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:53:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:53:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:53:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:53:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:46:29 2024 2024-04-24T22:53:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:46:29 2024 2024-04-24T22:53:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:46:29 2024 2024-04-24T22:53:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:53:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:53:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:53:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 1cecaa75-0bd6-483a-abc1-814cd6c8b99b 2024-04-24T22:53:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:53:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_44 snap 2024-04-24T22:53:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T22:53:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:53:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:53:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_44.export 2024-04-24T22:53:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_44.export 2024-04-24T22:53:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_44.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_44.export 2024-04-24T22:53:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_44.export 2024-04-24T22:53:10.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:53:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_44.export 2024-04-24T22:53:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:53:10.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_44.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_44.export 2024-04-24T22:53:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_44.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_44.export 2024-04-24T22:53:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:53:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_45 2024-04-24T22:53:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_45 snap 2024-04-24T22:53:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T22:53:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:53:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:53:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2024-04-24T22:53:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T22:53:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2024-04-24T22:53:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T22:53:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:53:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:53:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:53:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:53:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2024-04-24T22:53:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T22:53:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:53:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:53:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:53:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2024-04-24T22:53:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:53:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:53:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:53:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:53:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:53:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:53:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:53:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:53:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:53:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2024-04-24T22:53:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:53:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:53:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:53:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:53:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:53:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:53:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:53:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:53:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:53:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:53:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:53:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:53:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:53:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:53:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:53:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:53:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:53:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:53:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:53:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2024-04-24T22:53:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T22:53:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:53:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:53:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2024-04-24T22:53:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:53:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:53:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:53:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:53:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:53:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:53:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:53:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:53:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:53:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:53:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:53:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:53:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:53:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:53:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:53:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:53:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:53:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:53:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:53:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2024-04-24T22:53:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T22:53:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:53:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:53:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2024-04-24T22:53:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:53:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:53:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:53:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:53:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:53:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:53:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:53:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:53:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:53:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:53:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:53:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:53:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:53:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:53:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:53:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:53:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:53:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:53:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:53:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2024-04-24T22:53:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-04-24T22:53:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:53:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:53:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_45 2024-04-24T22:53:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:53:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:53:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:53:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:53:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:53:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:53:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:53:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:53:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:53:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:53:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:53:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:53:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:53:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:53:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:53:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:53:13.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:53:13.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:53:13.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:53:13.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_45 2024-04-24T22:53:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:53:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:53:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:53:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:53:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:53:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:53:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:53:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2024-04-24T22:53:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:53:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T22:53:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:53:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:53:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2024-04-24T22:53:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:53:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T22:53:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_45.status 2024-04-24T22:53:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_45.status 2024-04-24T22:53:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:53:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_45.status 2024-04-24T22:53:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_45.status 2024-04-24T22:53:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_45.status 2024-04-24T22:53:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:53:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_45.status 2024-04-24T22:53:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2024-04-24T22:53:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T22:53:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2024-04-24T22:53:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 207 2024-04-24T22:53:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_45.status 2024-04-24T22:53:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T22:53:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_45.status 2024-04-24T22:53:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2024-04-24T22:53:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:53:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=207 2024-04-24T22:53:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:53:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:53:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_45.status 2024-04-24T22:53:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T22:53:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_45.status 2024-04-24T22:53:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-04-24T22:53:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:53:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:14.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:53:14.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:53:14.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:53:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:53:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 207 = '' 2024-04-24T22:53:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:53:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_45.status 2024-04-24T22:53:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T22:53:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_45.status 2024-04-24T22:53:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-04-24T22:53:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:53:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:53:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:53:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:53:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:53:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 207 = '' 2024-04-24T22:53:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:53:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_45.status 2024-04-24T22:53:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T22:53:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_45.status 2024-04-24T22:53:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-04-24T22:53:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:15.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=207 2024-04-24T22:53:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 207 = 207 2024-04-24T22:53:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2024-04-24T22:53:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T22:53:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:53:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:53:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:53:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:53:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2024-04-24T22:53:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_45': 2024-04-24T22:53:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:53:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:53:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:53:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dcfac5173c 2024-04-24T22:53:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dcfac5173c 2024-04-24T22:53:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:53:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:53:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:53:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:53:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:46:37 2024 2024-04-24T22:53:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:46:37 2024 2024-04-24T22:53:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:46:37 2024 2024-04-24T22:53:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:53:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:53:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:53:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 94475b78-a054-4f7b-8225-9f06e6be39b9 2024-04-24T22:53:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:53:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_45 snap 2024-04-24T22:53:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T22:53:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:53:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:53:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_45.export 2024-04-24T22:53:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_45.export 2024-04-24T22:53:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_45.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_45.export 2024-04-24T22:53:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_45.export 2024-04-24T22:53:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:53:17.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_45.export 2024-04-24T22:53:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:53:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_45.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_45.export 2024-04-24T22:53:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_45.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_45.export 2024-04-24T22:53:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:53:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_46 2024-04-24T22:53:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_46 snap 2024-04-24T22:53:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T22:53:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:53:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:53:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2024-04-24T22:53:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T22:53:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2024-04-24T22:53:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T22:53:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:53:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:53:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:53:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:53:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2024-04-24T22:53:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T22:53:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:53:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:53:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:53:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2024-04-24T22:53:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:53:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:53:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:53:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:53:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:53:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:53:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:53:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:53:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:53:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2024-04-24T22:53:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:53:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:53:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:53:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:53:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:53:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:53:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:53:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:53:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:53:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:53:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:53:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:53:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:53:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:53:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:53:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:53:19.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:53:19.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:53:19.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:53:19.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2024-04-24T22:53:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T22:53:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:53:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:53:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2024-04-24T22:53:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:53:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:53:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:53:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:53:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:53:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:53:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:53:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:53:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:53:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:53:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:53:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:53:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:53:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:53:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:53:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:53:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:53:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:53:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:53:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2024-04-24T22:53:19.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T22:53:19.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:53:19.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:53:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_46 2024-04-24T22:53:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:53:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:53:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:53:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:53:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:53:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:53:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:53:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:53:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:53:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:53:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:53:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:53:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:53:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:53:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:53:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:53:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:53:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:53:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:53:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_46 2024-04-24T22:53:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-04-24T22:53:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:53:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:53:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_46 2024-04-24T22:53:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:53:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:53:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:53:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:53:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:53:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:53:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:53:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:53:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:53:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:53:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:53:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:53:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:53:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:53:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:53:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:53:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:53:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:53:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:53:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_46 2024-04-24T22:53:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:53:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:53:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:53:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:53:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:53:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:53:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:53:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2024-04-24T22:53:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:53:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T22:53:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:53:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:53:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2024-04-24T22:53:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:53:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T22:53:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_46.status 2024-04-24T22:53:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_46.status 2024-04-24T22:53:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:53:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_46.status 2024-04-24T22:53:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_46.status 2024-04-24T22:53:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_46.status 2024-04-24T22:53:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:53:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_46.status 2024-04-24T22:53:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2024-04-24T22:53:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T22:53:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2024-04-24T22:53:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 209 2024-04-24T22:53:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_46.status 2024-04-24T22:53:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T22:53:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_46.status 2024-04-24T22:53:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2024-04-24T22:53:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:20.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:53:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=209 2024-04-24T22:53:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:53:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:53:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_46.status 2024-04-24T22:53:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T22:53:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_46.status 2024-04-24T22:53:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-04-24T22:53:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:21.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=113 2024-04-24T22:53:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 209 = 113 2024-04-24T22:53:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:53:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_46.status 2024-04-24T22:53:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T22:53:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_46.status 2024-04-24T22:53:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-04-24T22:53:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:53:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:53:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:53:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:53:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:53:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 209 = '' 2024-04-24T22:53:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:53:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_46.status 2024-04-24T22:53:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T22:53:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_46.status 2024-04-24T22:53:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-04-24T22:53:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:53:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:53:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:53:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:53:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:53:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 209 = '' 2024-04-24T22:53:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T22:53:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_46.status 2024-04-24T22:53:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T22:53:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_46.status 2024-04-24T22:53:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-04-24T22:53:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=209 2024-04-24T22:53:24.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 209 = 209 2024-04-24T22:53:24.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:24.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2024-04-24T22:53:24.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:24.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:24.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T22:53:24.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:53:24.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:53:24.067 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:53:24.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:53:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2024-04-24T22:53:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_46': 2024-04-24T22:53:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:53:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:53:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:53:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc4f947717 2024-04-24T22:53:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc4f947717 2024-04-24T22:53:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:53:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:53:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:53:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:53:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:46:48 2024 2024-04-24T22:53:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:46:48 2024 2024-04-24T22:53:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:46:48 2024 2024-04-24T22:53:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:53:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:53:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:53:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: aaca0d68-80e6-4367-94cf-716faf3d8e67 2024-04-24T22:53:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:53:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_46 snap 2024-04-24T22:53:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T22:53:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:53:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:53:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_46.export 2024-04-24T22:53:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_46.export 2024-04-24T22:53:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_46.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_46.export 2024-04-24T22:53:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_46.export 2024-04-24T22:53:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:53:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_46.export 2024-04-24T22:53:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:53:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_46.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_46.export 2024-04-24T22:53:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_46.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_46.export 2024-04-24T22:53:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:53:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_47 2024-04-24T22:53:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_47 snap 2024-04-24T22:53:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T22:53:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:53:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:53:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2024-04-24T22:53:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T22:53:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2024-04-24T22:53:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T22:53:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:53:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:53:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:53:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:53:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2024-04-24T22:53:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T22:53:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:53:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:53:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:53:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2024-04-24T22:53:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:53:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:53:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:53:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:53:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:53:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:53:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:53:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:53:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:53:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-04-24T22:53:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:53:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:53:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:53:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:53:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:53:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:53:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:53:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:53:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:53:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:53:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:53:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:53:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:53:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:53:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:53:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:53:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:53:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:53:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:53:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-04-24T22:53:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-24T22:53:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:53:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:53:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2024-04-24T22:53:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:53:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:53:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:53:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:53:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:53:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:53:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:53:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:53:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:53:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:53:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:53:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:53:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:53:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:53:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:53:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:53:27.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:53:27.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:53:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:53:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2024-04-24T22:53:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-24T22:53:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:53:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:53:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2024-04-24T22:53:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:53:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:53:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:53:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:53:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:53:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:53:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:53:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:53:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:53:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:53:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:53:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:53:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:53:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:53:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:53:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:53:27.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:53:27.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:53:27.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:53:27.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2024-04-24T22:53:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-04-24T22:53:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:53:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:53:28.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:28.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_47 2024-04-24T22:53:28.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:53:28.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:28.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:28.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:53:28.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:53:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:53:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:53:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:53:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:53:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:53:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:53:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:53:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:53:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:53:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:53:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:53:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:53:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:53:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:53:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:53:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:53:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_47 2024-04-24T22:53:28.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:53:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:53:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:53:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:53:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:53:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:53:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:53:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2024-04-24T22:53:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:53:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T22:53:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:53:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:53:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2024-04-24T22:53:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:53:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T22:53:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_47.status 2024-04-24T22:53:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_47.status 2024-04-24T22:53:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:53:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_47.status 2024-04-24T22:53:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_47.status 2024-04-24T22:53:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_47.status 2024-04-24T22:53:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:53:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_47.status 2024-04-24T22:53:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2024-04-24T22:53:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T22:53:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2024-04-24T22:53:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 211 2024-04-24T22:53:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_47.status 2024-04-24T22:53:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T22:53:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_47.status 2024-04-24T22:53:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2024-04-24T22:53:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:53:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=211 2024-04-24T22:53:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:53:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:53:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_47.status 2024-04-24T22:53:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T22:53:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_47.status 2024-04-24T22:53:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-04-24T22:53:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:29.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:29.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=114 2024-04-24T22:53:29.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 211 = 114 2024-04-24T22:53:29.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:29.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:53:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_47.status 2024-04-24T22:53:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T22:53:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_47.status 2024-04-24T22:53:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-04-24T22:53:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=114 2024-04-24T22:53:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 211 = 114 2024-04-24T22:53:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:53:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_47.status 2024-04-24T22:53:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T22:53:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_47.status 2024-04-24T22:53:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-04-24T22:53:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:53:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:53:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:53:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:53:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:53:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 211 = '' 2024-04-24T22:53:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T22:53:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_47.status 2024-04-24T22:53:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T22:53:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_47.status 2024-04-24T22:53:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-04-24T22:53:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=211 2024-04-24T22:53:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 211 = 211 2024-04-24T22:53:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2024-04-24T22:53:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T22:53:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:53:31.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:53:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:53:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:53:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2024-04-24T22:53:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_47': 2024-04-24T22:53:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:53:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:53:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:53:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dcb01c1517 2024-04-24T22:53:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dcb01c1517 2024-04-24T22:53:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:53:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:53:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:53:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:53:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:46:54 2024 2024-04-24T22:53:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:46:54 2024 2024-04-24T22:53:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:46:54 2024 2024-04-24T22:53:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:53:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:53:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:53:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: b943dd73-ded7-434d-8abb-ad7480804db1 2024-04-24T22:53:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:53:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_47 snap 2024-04-24T22:53:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T22:53:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:53:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:53:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_47.export 2024-04-24T22:53:33.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_47.export 2024-04-24T22:53:33.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_47.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_47.export 2024-04-24T22:53:33.011 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_47.export 2024-04-24T22:53:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:53:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_47.export 2024-04-24T22:53:33.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:53:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_47.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_47.export 2024-04-24T22:53:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_47.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_47.export 2024-04-24T22:53:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:53:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_48 2024-04-24T22:53:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_48 snap 2024-04-24T22:53:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T22:53:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:53:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:53:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2024-04-24T22:53:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T22:53:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2024-04-24T22:53:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T22:53:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:53:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:53:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:53:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:53:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2024-04-24T22:53:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T22:53:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:53:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:53:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:53:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2024-04-24T22:53:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:53:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:53:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:53:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:53:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:53:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:53:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:53:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:53:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:53:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2024-04-24T22:53:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:53:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:53:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:53:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:53:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:53:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:53:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:53:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:53:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:53:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:53:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:53:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:53:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:53:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:53:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:53:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:53:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:53:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:53:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:53:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2024-04-24T22:53:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-04-24T22:53:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:53:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:53:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2024-04-24T22:53:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:53:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:53:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:53:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:53:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:53:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:53:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:53:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:53:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:53:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:53:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:53:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:53:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:53:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:53:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:53:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:53:36.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:53:36.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:53:36.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:53:36.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2024-04-24T22:53:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-04-24T22:53:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:53:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:53:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_48 2024-04-24T22:53:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:53:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:53:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:53:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:53:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:53:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:53:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:53:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:53:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:53:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:53:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:53:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:53:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:53:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:53:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:53:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:53:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:53:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:53:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:53:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_48 2024-04-24T22:53:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-04-24T22:53:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:53:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:53:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_48 2024-04-24T22:53:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:53:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:53:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:53:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:53:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:53:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:53:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:53:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:53:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:53:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:53:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:53:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:53:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:53:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:53:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:53:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:53:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:53:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:53:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:53:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_48 2024-04-24T22:53:36.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:53:36.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:53:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:53:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:53:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:53:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:53:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:53:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2024-04-24T22:53:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:53:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T22:53:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:53:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:53:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2024-04-24T22:53:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:53:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T22:53:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_48.status 2024-04-24T22:53:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_48.status 2024-04-24T22:53:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:53:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_48.status 2024-04-24T22:53:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_48.status 2024-04-24T22:53:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_48.status 2024-04-24T22:53:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:53:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_48.status 2024-04-24T22:53:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2024-04-24T22:53:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T22:53:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2024-04-24T22:53:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 213 2024-04-24T22:53:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_48.status 2024-04-24T22:53:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T22:53:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_48.status 2024-04-24T22:53:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2024-04-24T22:53:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:53:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=213 2024-04-24T22:53:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:53:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:53:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_48.status 2024-04-24T22:53:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T22:53:37.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_48.status 2024-04-24T22:53:37.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-04-24T22:53:37.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=115 2024-04-24T22:53:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 213 = 115 2024-04-24T22:53:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:53:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_48.status 2024-04-24T22:53:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T22:53:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_48.status 2024-04-24T22:53:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-04-24T22:53:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:53:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:53:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:53:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:53:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:53:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 213 = '' 2024-04-24T22:53:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:53:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_48.status 2024-04-24T22:53:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T22:53:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_48.status 2024-04-24T22:53:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-04-24T22:53:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=213 2024-04-24T22:53:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 213 = 213 2024-04-24T22:53:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2024-04-24T22:53:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T22:53:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:53:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:53:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:53:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:53:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2024-04-24T22:53:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_48': 2024-04-24T22:53:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:53:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:53:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:53:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc6a89f40b 2024-04-24T22:53:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc6a89f40b 2024-04-24T22:53:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:53:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:53:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:53:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:53:39.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:47:02 2024 2024-04-24T22:53:39.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:47:02 2024 2024-04-24T22:53:39.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:47:02 2024 2024-04-24T22:53:39.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:53:39.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:53:39.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:53:39.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 4b92c784-d7e2-461f-9fc8-08abc3ca10c8 2024-04-24T22:53:39.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:53:39.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_48 snap 2024-04-24T22:53:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T22:53:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:53:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:53:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_48.export 2024-04-24T22:53:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_48.export 2024-04-24T22:53:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_48.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_48.export 2024-04-24T22:53:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_48.export 2024-04-24T22:53:40.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:53:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_48.export 2024-04-24T22:53:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:53:40.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_48.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_48.export 2024-04-24T22:53:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_48.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_48.export 2024-04-24T22:53:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:53:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_49 2024-04-24T22:53:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_49 snap 2024-04-24T22:53:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T22:53:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:53:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:53:42.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2024-04-24T22:53:42.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:42.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:42.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T22:53:42.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2024-04-24T22:53:42.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:42.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:42.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T22:53:42.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:53:42.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:53:42.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:53:42.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:53:42.678 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4607570614278419) > weight (0.25) 2024-04-24T22:53:42.679 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9463647727380283) > weight (0.25) 2024-04-24T22:53:42.679 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-04-24T22:53:42.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-04-24T22:53:42.679 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-04-24T22:53:42.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T22:53:42.679+0000 7fd8f9b8c700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 62093) UID: 1000 2024-04-24T22:53:42.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:53:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2024-04-24T22:53:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T22:53:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:53:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:53:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:53:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2024-04-24T22:53:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:53:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:53:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:53:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:53:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:53:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:53:43.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:53:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:53:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:53:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2024-04-24T22:53:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:53:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:53:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:53:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:53:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:53:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:53:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:53:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:53:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:53:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:53:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:53:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:53:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:53:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:53:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:53:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:53:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:53:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:53:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:53:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2024-04-24T22:53:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-04-24T22:53:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:53:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:53:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2024-04-24T22:53:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:53:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:53:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:53:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:53:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:53:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:53:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:53:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:53:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:53:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:53:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:53:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:53:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:53:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:53:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:53:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:53:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:53:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:53:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:53:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2024-04-24T22:53:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-04-24T22:53:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:53:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:53:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_49 2024-04-24T22:53:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:53:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:53:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:53:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:53:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:53:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:53:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:53:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:53:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:53:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:53:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:53:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:53:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:53:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:53:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:53:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:53:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:53:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:53:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:53:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_49 2024-04-24T22:53:43.680 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-04-24T22:53:43.681 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:53:43.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:53:43.681 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-04-24T22:53:43.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-24T22:53:43.681 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-24T22:53:43.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-24T22:53:43.684 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2024-04-24T22:53:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-04-24T22:53:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_49 2024-04-24T22:53:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:53:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:53:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:53:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:53:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:53:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:53:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:53:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:53:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:53:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:53:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:53:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:53:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:53:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:53:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:53:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:53:43.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:53:43.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:53:43.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:53:43.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_49 2024-04-24T22:53:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:53:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:53:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:53:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:53:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:53:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:53:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:53:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2024-04-24T22:53:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:53:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T22:53:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:53:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:53:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2024-04-24T22:53:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:53:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T22:53:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_49.status 2024-04-24T22:53:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_49.status 2024-04-24T22:53:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:53:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_49.status 2024-04-24T22:53:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_49.status 2024-04-24T22:53:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_49.status 2024-04-24T22:53:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:53:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_49.status 2024-04-24T22:53:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2024-04-24T22:53:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T22:53:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2024-04-24T22:53:44.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 215 2024-04-24T22:53:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_49.status 2024-04-24T22:53:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T22:53:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_49.status 2024-04-24T22:53:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2024-04-24T22:53:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:53:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=215 2024-04-24T22:53:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:53:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:53:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_49.status 2024-04-24T22:53:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T22:53:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_49.status 2024-04-24T22:53:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-04-24T22:53:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:44.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:53:44.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:44.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:53:44.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:53:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:53:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:53:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 215 = '' 2024-04-24T22:53:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:53:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_49.status 2024-04-24T22:53:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T22:53:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_49.status 2024-04-24T22:53:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-04-24T22:53:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:53:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:53:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:53:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:53:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:53:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 215 = '' 2024-04-24T22:53:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:53:45.542 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7912239064594128) > weight (0.25) 2024-04-24T22:53:45.542 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8242593952980141) > weight (0.25) 2024-04-24T22:53:45.542 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.585636025385124) > weight (0.25) 2024-04-24T22:53:45.542 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.4690430922199704) > weight (0.25) 2024-04-24T22:53:45.542 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2024-04-24T22:53:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_49.status 2024-04-24T22:53:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T22:53:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_49.status 2024-04-24T22:53:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-04-24T22:53:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=215 2024-04-24T22:53:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 215 = 215 2024-04-24T22:53:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2024-04-24T22:53:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T22:53:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:53:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:53:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:53:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:53:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2024-04-24T22:53:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_49': 2024-04-24T22:53:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:53:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:53:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:53:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dc9564d9fa 2024-04-24T22:53:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dc9564d9fa 2024-04-24T22:53:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:53:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:53:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:53:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:53:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:47:10 2024 2024-04-24T22:53:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:47:10 2024 2024-04-24T22:53:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:47:10 2024 2024-04-24T22:53:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:53:46.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:53:46.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:53:46.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: ae301ebb-cfbe-4565-b97c-7f626fd5375e 2024-04-24T22:53:46.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:53:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_49 snap 2024-04-24T22:53:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T22:53:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:53:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:53:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_49.export 2024-04-24T22:53:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_49.export 2024-04-24T22:53:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_49.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_49.export 2024-04-24T22:53:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_49.export 2024-04-24T22:53:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:53:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_49.export 2024-04-24T22:53:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:53:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_49.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_49.export 2024-04-24T22:53:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_49.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_49.export 2024-04-24T22:53:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:53:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_50 2024-04-24T22:53:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_snap cluster2 mirror image_50 snap 2024-04-24T22:53:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T22:53:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:53:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2024-04-24T22:53:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2024-04-24T22:53:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T22:53:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2024-04-24T22:53:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T22:53:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=started 2024-04-24T22:53:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:53:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:53:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:53:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2024-04-24T22:53:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T22:53:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=started 2024-04-24T22:53:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_result 2024-04-24T22:53:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=stopped 2024-04-24T22:53:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2024-04-24T22:53:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -i state 2024-04-24T22:53:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_instance=cluster1 2024-04-24T22:53:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:53:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance=cluster1 2024-04-24T22:53:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local loop_instance 2024-04-24T22:53:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T22:53:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sleep 0 2024-04-24T22:53:49.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-04-24T22:53:49.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ seq 0 3 2024-04-24T22:53:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-04-24T22:53:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:0 2024-04-24T22:53:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T22:53:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:0 2024-04-24T22:53:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:53:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T22:53:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=0 2024-04-24T22:53:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:53:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-04-24T22:53:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:0 2024-04-24T22:53:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T22:53:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:0 2024-04-24T22:53:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=0 2024-04-24T22:53:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T22:53:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=0 2024-04-24T22:53:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=0 2024-04-24T22:53:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T22:53:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:53:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:53:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T22:53:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-04-24T22:53:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T22:53:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:53:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:53:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2024-04-24T22:53:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:1 2024-04-24T22:53:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-04-24T22:53:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:1 2024-04-24T22:53:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=1 2024-04-24T22:53:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 1 = cluster1:1 ']' 2024-04-24T22:53:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=1 2024-04-24T22:53:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:53:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-04-24T22:53:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:1 2024-04-24T22:53:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-04-24T22:53:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:1 2024-04-24T22:53:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=1 2024-04-24T22:53:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 1 = cluster1:1 ']' 2024-04-24T22:53:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=1 2024-04-24T22:53:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=1 2024-04-24T22:53:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-04-24T22:53:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:53:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:53:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-04-24T22:53:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2024-04-24T22:53:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T22:53:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:53:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:53:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_50 2024-04-24T22:53:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:2 2024-04-24T22:53:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-04-24T22:53:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:2 2024-04-24T22:53:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=2 2024-04-24T22:53:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 2 = cluster1:2 ']' 2024-04-24T22:53:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=2 2024-04-24T22:53:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:53:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-04-24T22:53:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:2 2024-04-24T22:53:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-04-24T22:53:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:2 2024-04-24T22:53:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=2 2024-04-24T22:53:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 2 = cluster1:2 ']' 2024-04-24T22:53:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=2 2024-04-24T22:53:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=2 2024-04-24T22:53:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-04-24T22:53:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:53:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:53:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-04-24T22:53:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_50 2024-04-24T22:53:49.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-04-24T22:53:49.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd mirror status : get status for rbd mirror 2024-04-24T22:53:49.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:admin_socket: invalid command 2024-04-24T22:53:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T22:53:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_50 2024-04-24T22:53:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1:3 2024-04-24T22:53:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ shift 2024-04-24T22:53:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance 2024-04-24T22:53:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-04-24T22:53:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local val=cluster1:3 2024-04-24T22:53:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster_var_name=cluster 2024-04-24T22:53:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local instance_var_name=instance 2024-04-24T22:53:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:53:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=3 2024-04-24T22:53:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ '[' 3 = cluster1:3 ']' 2024-04-24T22:53:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval cluster=cluster1 2024-04-24T22:53:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ eval instance=3 2024-04-24T22:53:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:53:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-04-24T22:53:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local local_cluster=cluster1:3 2024-04-24T22:53:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster=cluster1 2024-04-24T22:53:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance 2024-04-24T22:53:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-04-24T22:53:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local val=cluster1:3 2024-04-24T22:53:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local cluster_var_name=local_cluster 2024-04-24T22:53:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ local instance_var_name=instance 2024-04-24T22:53:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ cluster=cluster1 2024-04-24T22:53:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ instance=3 2024-04-24T22:53:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ '[' 3 = cluster1:3 ']' 2024-04-24T22:53:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval local_cluster=cluster1 2024-04-24T22:53:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ local_cluster=cluster1 2024-04-24T22:53:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ eval instance=3 2024-04-24T22:53:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ instance=3 2024-04-24T22:53:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-04-24T22:53:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:53:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:53:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-04-24T22:53:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_50 2024-04-24T22:53:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ return 0 2024-04-24T22:53:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ status_result=' "state": "Replaying"' 2024-04-24T22:53:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo ' "state": "Replaying"' 2024-04-24T22:53:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ grep -i Replaying 2024-04-24T22:53:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: "state": "Replaying" 2024-04-24T22:53:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=started 2024-04-24T22:53:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test started = started 2024-04-24T22:53:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2024-04-24T22:53:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:53:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T22:53:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = journal ']' 2024-04-24T22:53:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' snapshot = snapshot ']' 2024-04-24T22:53:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2024-04-24T22:53:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_cluster=cluster1 2024-04-24T22:53:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T22:53:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster2-mirror-image_50.status 2024-04-24T22:53:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster2-mirror-image_50.status 2024-04-24T22:53:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:53:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local status_log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_50.status 2024-04-24T22:53:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ mkfname cluster1-mirror-image_50.status 2024-04-24T22:53:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ echo cluster1-mirror-image_50.status 2024-04-24T22:53:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -e 's|[/ ]|_|g' 2024-04-24T22:53:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local local_status_log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_50.status 2024-04-24T22:53:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2024-04-24T22:53:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T22:53:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2024-04-24T22:53:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Snapshot ID: 217 2024-04-24T22:53:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_50.status 2024-04-24T22:53:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T22:53:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_50.status 2024-04-24T22:53:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2024-04-24T22:53:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-04-24T22:53:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snapshot_id=217 2024-04-24T22:53:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:53:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.2 2024-04-24T22:53:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_50.status 2024-04-24T22:53:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T22:53:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_50.status 2024-04-24T22:53:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-04-24T22:53:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:50.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=117 2024-04-24T22:53:50.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 217 = 117 2024-04-24T22:53:50.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:50.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.4 2024-04-24T22:53:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_50.status 2024-04-24T22:53:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T22:53:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_50.status 2024-04-24T22:53:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-04-24T22:53:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=117 2024-04-24T22:53:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 217 = 117 2024-04-24T22:53:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.8 2024-04-24T22:53:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_50.status 2024-04-24T22:53:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T22:53:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_50.status 2024-04-24T22:53:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-04-24T22:53:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:53:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:-:1.1: Document is empty 2024-04-24T22:53:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: 2024-04-24T22:53:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:^ 2024-04-24T22:53:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id= 2024-04-24T22:53:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 217 = '' 2024-04-24T22:53:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-04-24T22:53:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1.6 2024-04-24T22:53:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_50.status 2024-04-24T22:53:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T22:53:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local log=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_50.status 2024-04-24T22:53:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-04-24T22:53:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-04-24T22:53:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-04-24T22:53:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local primary_snapshot_id=217 2024-04-24T22:53:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 217 = 217 2024-04-24T22:53:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2024-04-24T22:53:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:53:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T22:53:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:53:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:53:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-04-24T22:53:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T22:53:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2024-04-24T22:53:54.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_50': 2024-04-24T22:53:54.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 128 MiB in 32 objects 2024-04-24T22:53:54.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:53:54.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 2 2024-04-24T22:53:54.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 10dca2538cf6 2024-04-24T22:53:54.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.10dca2538cf6 2024-04-24T22:53:54.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:53:54.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-04-24T22:53:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:53:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:53:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:47:16 2024 2024-04-24T22:53:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:47:16 2024 2024-04-24T22:53:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:47:16 2024 2024-04-24T22:53:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: protected: False 2024-04-24T22:53:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:53:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:53:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 19c08b8e-0c4b-4471-adf6-b6a40dd52550 2024-04-24T22:53:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: false 2024-04-24T22:53:54.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:54.904 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ compare_image_snaps mirror image_50 snap 2024-04-24T22:53:54.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:54.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T22:53:54.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap_name=snap 2024-04-24T22:53:54.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local ret=0 2024-04-24T22:53:54.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local rmt_export=/tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_50.export 2024-04-24T22:53:54.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local loc_export=/tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_50.export 2024-04-24T22:53:54.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_50.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_50.export 2024-04-24T22:53:54.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_50.export 2024-04-24T22:53:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:53:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_50.export 2024-04-24T22:53:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-04-24T22:53:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cmp /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_50.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_50.export 2024-04-24T22:53:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -f /tmp/tmp.DCvHjwkiaM/cluster2-mirror-image_50.export /tmp/tmp.DCvHjwkiaM/cluster1-mirror-image_50.export 2024-04-24T22:53:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ testlog 'TEST: delete many images' 2024-04-24T22:53:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ tee -a /tmp/tmp.DCvHjwkiaM/rbd-mirror.test.log 2024-04-24T22:53:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ date '+%F %T' 2024-04-24T22:53:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo 2024-04-24 22:53:55 TEST: delete many images 2024-04-24T22:53:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:2024-04-24 22:53:55 TEST: delete many images 2024-04-24T22:53:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 50 2024-04-24T22:53:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:53:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_1 2024-04-24T22:53:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2024-04-24T22:53:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:53:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:53:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2024-04-24T22:53:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:53:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_1 2024-04-24T22:53:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:53:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:53:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:53:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_1 2024-04-24T22:53:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:53:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-04-24T22:53:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-04-24T22:53:55.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:53:55.777+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2aeb3d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:53:55.778 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:53:55.777+0000 7fecff3a9700 -1 rbd::mirror::ImageReplayer: 0x557d27914c80 [3/a0aa7e18-e06a-4e3c-b4d7-6bdccb4d22a8] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:53:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:53:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:53:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:53:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_2 2024-04-24T22:53:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2024-04-24T22:53:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:53:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:53:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2024-04-24T22:53:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:53:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_2 2024-04-24T22:53:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:53:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:53:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:53:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_2 2024-04-24T22:53:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:53:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:53:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:53:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-04-24T22:53:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-04-24T22:53:58.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:53:58.810+0000 7fecf939d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d27945ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:53:58.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:53:58.810+0000 7fecffbaa700 -1 rbd::mirror::ImageReplayer: 0x557d2ad58780 [3/9d47d6f5-f040-4915-aac4-a0dd1e04cc47] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:54:00.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:54:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:54:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:54:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_3 2024-04-24T22:54:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2024-04-24T22:54:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:54:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:54:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2024-04-24T22:54:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:54:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_3 2024-04-24T22:54:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:54:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:54:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:54:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_3 2024-04-24T22:54:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:54:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-04-24T22:54:01.716 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-04-24T22:54:01.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T22:54:01.797+0000 7f4a5070b700 -1 rbd::mirror::ImageReplayer: 0x55dd01954280 [3/e799308a-a04d-4e8c-b6d0-1485ec8486a2] start: already running 2024-04-24T22:54:01.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T22:54:01.797+0000 7f4a5070b700 -1 rbd::mirror::ImageReplayer: 0x55dd01954280 [3/e799308a-a04d-4e8c-b6d0-1485ec8486a2] start: already running 2024-04-24T22:54:01.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:01.829+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d27944000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:54:01.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:01.829+0000 7fecffbaa700 -1 rbd::mirror::ImageReplayer: 0x557d2ad58a00 [3/e799308a-a04d-4e8c-b6d0-1485ec8486a2] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:54:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:54:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:54:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:54:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_4 2024-04-24T22:54:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2024-04-24T22:54:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:54:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:54:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2024-04-24T22:54:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:54:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_4 2024-04-24T22:54:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:54:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:54:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:54:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_4 2024-04-24T22:54:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:54:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-04-24T22:54:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-04-24T22:54:04.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:04.374+0000 7fecf939d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2aeb3040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:54:04.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:04.374+0000 7fecff3a9700 -1 rbd::mirror::ImageReplayer: 0x557d27915900 [3/455d42c6-ae74-4f65-ae3f-0405438be886] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:54:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:54:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:54:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:54:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_5 2024-04-24T22:54:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2024-04-24T22:54:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:54:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:54:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2024-04-24T22:54:07.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:54:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_5 2024-04-24T22:54:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:54:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:54:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:54:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_5 2024-04-24T22:54:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:54:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-04-24T22:54:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-04-24T22:54:07.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:07.383+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2b302000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:54:07.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:07.383+0000 7fecff3a9700 -1 rbd::mirror::ImageReplayer: 0x557d2ad59680 [3/6fb19b55-9e60-4b51-937c-bfff39ac7c68] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:54:09.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:54:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:54:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:54:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_6 2024-04-24T22:54:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2024-04-24T22:54:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T22:54:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:54:09.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2024-04-24T22:54:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:54:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_6 2024-04-24T22:54:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T22:54:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:54:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:54:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_6 2024-04-24T22:54:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T22:54:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-04-24T22:54:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-04-24T22:54:10.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:10.388+0000 7fecf939d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d291d5d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:54:10.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:10.388+0000 7fecffbaa700 -1 rbd::mirror::ImageReplayer: 0x557d2ad59900 [3/059ce48e-b70a-434e-9dd3-e955c7a954da] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:54:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:54:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:54:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:54:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_7 2024-04-24T22:54:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2024-04-24T22:54:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T22:54:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:54:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2024-04-24T22:54:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:54:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_7 2024-04-24T22:54:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T22:54:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:54:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:54:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_7 2024-04-24T22:54:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T22:54:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-04-24T22:54:13.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-04-24T22:54:13.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:13.402+0000 7fecf939d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2ab82d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:54:13.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:13.402+0000 7fecff3a9700 -1 rbd::mirror::ImageReplayer: 0x557d27915680 [3/3cf64901-3ddb-4c2f-aca5-336fbdf69cfe] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:54:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:54:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:54:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:54:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_8 2024-04-24T22:54:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2024-04-24T22:54:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T22:54:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:54:15.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2024-04-24T22:54:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:54:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_8 2024-04-24T22:54:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T22:54:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:54:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:54:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_8 2024-04-24T22:54:16.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:16.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:16.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T22:54:16.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-04-24T22:54:16.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-04-24T22:54:16.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:16.420+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d291d4820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:54:16.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:16.420+0000 7fecffbaa700 -1 rbd::mirror::ImageReplayer: 0x557d27915b80 [3/8f5f4362-e6f1-477d-bce4-f3ab1a7caee3] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:54:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:54:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:54:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:54:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_9 2024-04-24T22:54:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2024-04-24T22:54:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T22:54:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:54:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2024-04-24T22:54:19.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:54:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_9 2024-04-24T22:54:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T22:54:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:54:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:54:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_9 2024-04-24T22:54:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T22:54:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-04-24T22:54:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-04-24T22:54:19.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:19.440+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d291d49c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:54:19.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:19.440+0000 7fecffbaa700 -1 rbd::mirror::ImageReplayer: 0x557d2ae1e780 [3/59c6ba05-b01f-4e71-ad9c-960605496d0d] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:54:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:54:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:54:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:54:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_10 2024-04-24T22:54:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2024-04-24T22:54:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T22:54:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:54:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2024-04-24T22:54:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:54:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_10 2024-04-24T22:54:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T22:54:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:54:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:54:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_10 2024-04-24T22:54:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T22:54:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-04-24T22:54:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-04-24T22:54:22.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:22.649+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2aa45a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:54:22.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:22.649+0000 7fecff3a9700 -1 rbd::mirror::ImageReplayer: 0x557d2ae1e000 [3/020b3f22-a97b-4ec7-aca6-b7942750a5f1] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:54:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:54:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:54:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:54:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_11 2024-04-24T22:54:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2024-04-24T22:54:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:24.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:24.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T22:54:24.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:54:24.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2024-04-24T22:54:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:54:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_11 2024-04-24T22:54:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T22:54:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:54:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:54:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_11 2024-04-24T22:54:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T22:54:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2024-04-24T22:54:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2024-04-24T22:54:25.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:25.355+0000 7fecf939d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d291d4b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:54:25.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:25.355+0000 7fecffbaa700 -1 rbd::mirror::ImageReplayer: 0x557d2ae1ea00 [3/8e1cc3c0-2ace-44c5-a115-11e73865c94e] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:54:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:54:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:54:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:54:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_12 2024-04-24T22:54:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2024-04-24T22:54:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T22:54:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:54:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2024-04-24T22:54:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:54:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_12 2024-04-24T22:54:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T22:54:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:54:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:54:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_12 2024-04-24T22:54:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T22:54:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2024-04-24T22:54:28.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2024-04-24T22:54:28.332 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:54:28.331+0000 7f0f4ef7b700 -1 reset not still connected to 0x561388b12f70 2024-04-24T22:54:28.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:28.394+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2aa45040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:54:28.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:28.394+0000 7fecff3a9700 -1 rbd::mirror::ImageReplayer: 0x557d2ae1e280 [3/95e4bc86-7500-4354-96f7-2ccfc5fb3d66] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:54:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:54:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:54:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:54:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_13 2024-04-24T22:54:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2024-04-24T22:54:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T22:54:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:54:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2024-04-24T22:54:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:54:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_13 2024-04-24T22:54:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T22:54:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:54:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:54:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_13 2024-04-24T22:54:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T22:54:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2024-04-24T22:54:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2024-04-24T22:54:31.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:31.393+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d291d4d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:54:31.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:31.393+0000 7fecffbaa700 -1 rbd::mirror::ImageReplayer: 0x557d2ae1e500 [3/4bd2c65f-f68d-4a16-b6d4-a7ab70837e25] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:54:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:54:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:54:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:54:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_14 2024-04-24T22:54:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2024-04-24T22:54:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T22:54:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:54:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2024-04-24T22:54:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:54:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_14 2024-04-24T22:54:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T22:54:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:54:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:54:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_14 2024-04-24T22:54:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T22:54:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2024-04-24T22:54:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2024-04-24T22:54:34.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:34.403+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d291d4340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:54:34.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:34.403+0000 7fecffbaa700 -1 rbd::mirror::ImageReplayer: 0x557d2ae1f680 [3/e99fde4e-a334-4491-9ca4-d28b703e6700] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:54:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:54:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:54:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:54:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_15 2024-04-24T22:54:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2024-04-24T22:54:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T22:54:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:54:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2024-04-24T22:54:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:54:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_15 2024-04-24T22:54:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T22:54:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:54:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:54:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_15 2024-04-24T22:54:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T22:54:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2024-04-24T22:54:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2024-04-24T22:54:37.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:37.420+0000 7fecf939d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d279456c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:54:37.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:37.420+0000 7fecff3a9700 -1 rbd::mirror::ImageReplayer: 0x557d2af5e000 [3/61a0be27-386b-497f-bf17-b0030cfd9593] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:54:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:54:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:54:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:54:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_16 2024-04-24T22:54:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2024-04-24T22:54:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T22:54:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:54:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2024-04-24T22:54:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:54:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_16 2024-04-24T22:54:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T22:54:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:54:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:54:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_16 2024-04-24T22:54:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T22:54:40.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2024-04-24T22:54:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2024-04-24T22:54:40.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:40.439+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2ace4340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:54:40.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:40.439+0000 7fecffbaa700 -1 rbd::mirror::ImageReplayer: 0x557d2af5e280 [3/73832a9a-0532-4696-9e85-d0aaea380e6c] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:54:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:54:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:54:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:54:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_17 2024-04-24T22:54:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2024-04-24T22:54:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T22:54:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:54:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2024-04-24T22:54:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:54:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_17 2024-04-24T22:54:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T22:54:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:54:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:54:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_17 2024-04-24T22:54:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T22:54:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2024-04-24T22:54:43.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2024-04-24T22:54:43.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:43.442+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2b303860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:54:43.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:43.442+0000 7fecff3a9700 -1 rbd::mirror::ImageReplayer: 0x557d2af5e500 [3/ea68e336-4843-42cb-80ec-9f3fa4c17a25] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:54:45.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:54:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:54:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:54:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_18 2024-04-24T22:54:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2024-04-24T22:54:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T22:54:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:54:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2024-04-24T22:54:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:54:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_18 2024-04-24T22:54:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T22:54:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:54:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:54:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_18 2024-04-24T22:54:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T22:54:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2024-04-24T22:54:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2024-04-24T22:54:46.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:46.453+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2aa3fba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:54:46.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:46.453+0000 7fecffbaa700 -1 rbd::mirror::ImageReplayer: 0x557d2ae1ec80 [3/e52f7b2d-1d65-436d-bbae-a4af8a5d1482] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:54:46.543 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7852700618782033) > weight (0.25) 2024-04-24T22:54:46.543 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5194317957729931) > weight (0.25) 2024-04-24T22:54:46.543 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-04-24T22:54:46.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-04-24T22:54:46.543 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-04-24T22:54:46.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T22:54:46.543+0000 7f7ef65eb700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 61888) UID: 1000 2024-04-24T22:54:46.745 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:54:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:54:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:54:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:54:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_19 2024-04-24T22:54:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2024-04-24T22:54:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T22:54:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:54:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2024-04-24T22:54:49.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:54:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_19 2024-04-24T22:54:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T22:54:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:54:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:54:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_19 2024-04-24T22:54:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T22:54:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2024-04-24T22:54:49.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2024-04-24T22:54:49.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:49.465+0000 7fecf939d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d29e0cea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:54:49.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:49.465+0000 7fecffbaa700 -1 rbd::mirror::ImageReplayer: 0x557d2ae1ef00 [3/38210526-ffd7-4449-be03-5e75b6c23176] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:54:49.685 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8511344364863404) > weight (0.25) 2024-04-24T22:54:49.685 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9794750330257335) > weight (0.25) 2024-04-24T22:54:49.685 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6514466241860519) > weight (0.25) 2024-04-24T22:54:49.685 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7718199536478877) > weight (0.25) 2024-04-24T22:54:49.685 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2024-04-24T22:54:51.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:54:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:54:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:54:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_20 2024-04-24T22:54:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2024-04-24T22:54:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T22:54:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:54:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2024-04-24T22:54:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:54:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_20 2024-04-24T22:54:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T22:54:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:54:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:54:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_20 2024-04-24T22:54:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T22:54:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2024-04-24T22:54:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2024-04-24T22:54:52.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:52.499+0000 7fecf939d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2aa451e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:54:52.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:52.499+0000 7fecff3a9700 -1 rbd::mirror::ImageReplayer: 0x557d2af5e780 [3/061306d6-3c6a-4f11-84ad-c1042b24a00e] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:54:53.545 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-04-24T22:54:53.545 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:54:53.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:54:53.545 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-04-24T22:54:53.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-04-24T22:54:53.546 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-04-24T22:54:53.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-04-24T22:54:53.549 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2024-04-24T22:54:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:54:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:54:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:54:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_21 2024-04-24T22:54:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2024-04-24T22:54:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T22:54:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:54:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2024-04-24T22:54:55.319 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:54:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_21 2024-04-24T22:54:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T22:54:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:54:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:54:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_21 2024-04-24T22:54:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T22:54:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2024-04-24T22:54:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2024-04-24T22:54:55.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:55.485+0000 7fecf939d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2b34b040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:54:55.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:55.485+0000 7fecffbaa700 -1 rbd::mirror::ImageReplayer: 0x557d2af5ea00 [3/90338b6b-d62c-42ea-9b32-905a5d005824] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:54:57.338 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:54:57.336+0000 7f0f4ef7b700 -1 reset not still connected to 0x56138781d2b0 2024-04-24T22:54:57.338 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:54:57.336+0000 7f0f4ef7b700 -1 reset not still connected to 0x561388b13380 2024-04-24T22:54:57.338 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:54:57.336+0000 7f0f4ef7b700 -1 reset not still connected to 0x56139637c000 2024-04-24T22:54:57.338 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:54:57.336+0000 7f0f4ef7b700 -1 reset not still connected to 0x56139637cd00 2024-04-24T22:54:57.345 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:54:57.339+0000 7f0f4ef7b700 -1 reset not still connected to 0x5613971f89c0 2024-04-24T22:54:57.346 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:54:57.339+0000 7f0f4ef7b700 -1 reset not still connected to 0x56139adb3790 2024-04-24T22:54:57.346 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:54:57.339+0000 7f0f4ef7b700 -1 reset not still connected to 0x56139b3fcf70 2024-04-24T22:54:57.346 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:54:57.339+0000 7f0f4ef7b700 -1 reset not still connected to 0x56139b3fd930 2024-04-24T22:54:57.346 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:54:57.339+0000 7f0f4ef7b700 -1 reset not still connected to 0x56139be86820 2024-04-24T22:54:57.346 INFO:tasks.ceph.cluster2.osd.2.smithi134.stderr:2024-04-24T22:54:57.339+0000 7f0f4ef7b700 -1 reset not still connected to 0x5613aa4f9d40 2024-04-24T22:54:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:54:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:54:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:54:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_22 2024-04-24T22:54:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2024-04-24T22:54:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T22:54:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:54:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2024-04-24T22:54:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:54:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_22 2024-04-24T22:54:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T22:54:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:54:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:54:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_22 2024-04-24T22:54:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:54:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:54:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T22:54:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2024-04-24T22:54:58.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2024-04-24T22:54:58.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:58.586+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d29e4c680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:54:58.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:54:58.586+0000 7fecff3a9700 -1 rbd::mirror::ImageReplayer: 0x557d2af5ec80 [3/6ab5e540-e8f6-406b-9bdd-00cf740c3964] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:55:00.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:55:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:55:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:55:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_23 2024-04-24T22:55:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2024-04-24T22:55:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T22:55:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:55:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2024-04-24T22:55:00.535 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:55:00.534+0000 7fd668aa0700 -1 reset not still connected to 0x56391a5ea8f0 2024-04-24T22:55:00.535 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:55:00.534+0000 7fd668aa0700 -1 reset not still connected to 0x56391a5ebee0 2024-04-24T22:55:00.535 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:55:00.534+0000 7fd668aa0700 -1 reset not still connected to 0x56391a644b60 2024-04-24T22:55:00.535 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:55:00.534+0000 7fd668aa0700 -1 reset not still connected to 0x56391a644f70 2024-04-24T22:55:00.536 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:55:00.535+0000 7fd668aa0700 -1 reset not still connected to 0x56391a7c95f0 2024-04-24T22:55:00.536 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:55:00.535+0000 7fd668aa0700 -1 reset not still connected to 0x56391a834820 2024-04-24T22:55:00.537 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:55:00.535+0000 7fd668aa0700 -1 reset not still connected to 0x56391f3d08f0 2024-04-24T22:55:00.537 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:55:00.535+0000 7fd668aa0700 -1 reset not still connected to 0x56391f3d1c70 2024-04-24T22:55:00.537 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:55:00.535+0000 7fd668aa0700 -1 reset not still connected to 0x56391f5e8a90 2024-04-24T22:55:00.537 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:55:00.535+0000 7fd668aa0700 -1 reset not still connected to 0x56391f85b6c0 2024-04-24T22:55:00.537 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:55:00.535+0000 7fd668aa0700 -1 reset not still connected to 0x56391f85bee0 2024-04-24T22:55:00.537 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:55:00.536+0000 7fd668aa0700 -1 reset not still connected to 0x56392a40dc70 2024-04-24T22:55:00.538 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:55:00.537+0000 7fd668aa0700 -1 reset not still connected to 0x56392acd80d0 2024-04-24T22:55:00.538 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:55:00.537+0000 7fd668aa0700 -1 reset not still connected to 0x56392af3ab60 2024-04-24T22:55:00.538 INFO:tasks.ceph.cluster1.osd.0.smithi080.stderr:2024-04-24T22:55:00.537+0000 7fd668aa0700 -1 reset not still connected to 0x56392c547110 2024-04-24T22:55:01.326 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:55:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_23 2024-04-24T22:55:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T22:55:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:55:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:55:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_23 2024-04-24T22:55:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T22:55:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2024-04-24T22:55:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2024-04-24T22:55:01.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:01.497+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2aeb4680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:55:01.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:01.498+0000 7fecffbaa700 -1 rbd::mirror::ImageReplayer: 0x557d291a4000 [3/c24b92fa-65bf-4177-9441-bcf5f91f92ff] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:55:03.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:55:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:55:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:55:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_24 2024-04-24T22:55:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2024-04-24T22:55:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T22:55:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:55:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2024-04-24T22:55:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:55:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_24 2024-04-24T22:55:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T22:55:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:55:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:55:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_24 2024-04-24T22:55:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T22:55:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2024-04-24T22:55:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2024-04-24T22:55:04.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:04.515+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2aa376c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:55:04.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:04.515+0000 7fecffbaa700 -1 rbd::mirror::ImageReplayer: 0x557d291a4280 [3/f25eb5c2-2f4d-4c6e-9c1e-1ee69ca3048a] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:55:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:55:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:55:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:55:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_25 2024-04-24T22:55:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2024-04-24T22:55:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T22:55:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:55:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2024-04-24T22:55:07.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:55:07.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_25 2024-04-24T22:55:07.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:07.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:07.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T22:55:07.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:55:07.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:55:07.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_25 2024-04-24T22:55:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T22:55:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2024-04-24T22:55:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2024-04-24T22:55:07.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:07.518+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2aeb4b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:55:07.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:07.518+0000 7fecffbaa700 -1 rbd::mirror::ImageReplayer: 0x557d2af5f400 [3/a6830866-682c-4c96-8b3b-75517871f35d] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:55:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:55:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:55:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:55:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_26 2024-04-24T22:55:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2024-04-24T22:55:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T22:55:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:55:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2024-04-24T22:55:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:55:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_26 2024-04-24T22:55:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T22:55:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:55:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:55:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_26 2024-04-24T22:55:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T22:55:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2024-04-24T22:55:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2024-04-24T22:55:10.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:10.383+0000 7fecf939d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2aa3e000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:55:10.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:10.383+0000 7fecff3a9700 -1 rbd::mirror::ImageReplayer: 0x557d291a4500 [3/f38aef72-1054-47c3-917d-19c1b6627e53] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:55:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:55:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:55:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:55:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_27 2024-04-24T22:55:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2024-04-24T22:55:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T22:55:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:55:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2024-04-24T22:55:13.223 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:55:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_27 2024-04-24T22:55:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T22:55:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:55:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:55:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_27 2024-04-24T22:55:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T22:55:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2024-04-24T22:55:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2024-04-24T22:55:13.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:13.395+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2aeb56c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:55:13.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:13.395+0000 7fecffbaa700 -1 rbd::mirror::ImageReplayer: 0x557d291a4780 [3/1143f5a5-897d-4fc6-98d2-c436f71170b7] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:55:15.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:55:15.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:55:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:55:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_28 2024-04-24T22:55:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2024-04-24T22:55:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T22:55:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:55:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2024-04-24T22:55:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:55:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_28 2024-04-24T22:55:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T22:55:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:55:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:55:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_28 2024-04-24T22:55:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T22:55:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2024-04-24T22:55:16.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2024-04-24T22:55:16.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:16.412+0000 7fecf939d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2aa3e1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:55:16.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:16.412+0000 7fecff3a9700 -1 rbd::mirror::ImageReplayer: 0x557d2af5f680 [3/0f1733b8-4c9d-4886-af25-8d2e8af19dd6] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:55:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:55:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:55:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:55:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_29 2024-04-24T22:55:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2024-04-24T22:55:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T22:55:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:55:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2024-04-24T22:55:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:55:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_29 2024-04-24T22:55:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T22:55:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:55:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:55:19.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_29 2024-04-24T22:55:19.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:19.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:19.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T22:55:19.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2024-04-24T22:55:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2024-04-24T22:55:19.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:19.416+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2aa37a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:55:19.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:19.416+0000 7fecff3a9700 -1 rbd::mirror::ImageReplayer: 0x557d291a4a00 [3/171b9277-b7b3-4a7f-ae2b-c3d8aca7eadf] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:55:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:55:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:55:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:55:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_30 2024-04-24T22:55:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2024-04-24T22:55:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T22:55:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:55:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2024-04-24T22:55:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:55:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_30 2024-04-24T22:55:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T22:55:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:55:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:55:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_30 2024-04-24T22:55:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T22:55:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2024-04-24T22:55:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2024-04-24T22:55:22.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:22.412+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2aa1c820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:55:22.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:22.412+0000 7fecffbaa700 -1 rbd::mirror::ImageReplayer: 0x557d2af5f900 [3/4cd18b79-d4e1-402a-89e5-a62956f139ed] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:55:24.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:55:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:55:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:55:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_31 2024-04-24T22:55:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2024-04-24T22:55:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T22:55:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:55:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2024-04-24T22:55:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:55:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_31 2024-04-24T22:55:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T22:55:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:55:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:55:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_31 2024-04-24T22:55:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T22:55:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2024-04-24T22:55:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2024-04-24T22:55:25.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:25.432+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2aa37ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:55:25.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:25.432+0000 7fecff3a9700 -1 rbd::mirror::ImageReplayer: 0x557d2af5fb80 [3/e73f5c22-d59e-4e21-8ef5-f803b29356cb] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:55:27.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:55:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:55:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:55:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_32 2024-04-24T22:55:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2024-04-24T22:55:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T22:55:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:55:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2024-04-24T22:55:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:55:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_32 2024-04-24T22:55:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T22:55:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:55:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:55:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_32 2024-04-24T22:55:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T22:55:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2024-04-24T22:55:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2024-04-24T22:55:28.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:28.398+0000 7fecf939d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d29e4c9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:55:28.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:28.398+0000 7fecffbaa700 -1 rbd::mirror::ImageReplayer: 0x557d291a5180 [3/f185f517-ef74-4863-98b2-5700a59468b6] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:55:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:55:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:55:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:55:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_33 2024-04-24T22:55:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2024-04-24T22:55:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T22:55:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:55:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2024-04-24T22:55:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:55:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_33 2024-04-24T22:55:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T22:55:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:55:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:55:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_33 2024-04-24T22:55:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T22:55:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2024-04-24T22:55:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2024-04-24T22:55:31.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:31.468+0000 7fecf939d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d29e0c1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:55:31.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:31.468+0000 7fecff3a9700 -1 rbd::mirror::ImageReplayer: 0x557d291a5400 [3/9907f064-0835-4f4a-86eb-5688b009e4d7] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:55:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:55:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:55:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:55:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_34 2024-04-24T22:55:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2024-04-24T22:55:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T22:55:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:55:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2024-04-24T22:55:34.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:55:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_34 2024-04-24T22:55:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T22:55:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:55:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:55:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_34 2024-04-24T22:55:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T22:55:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2024-04-24T22:55:34.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2024-04-24T22:55:34.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:34.474+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d27971040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:55:34.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:34.474+0000 7fecff3a9700 -1 rbd::mirror::ImageReplayer: 0x557d291a5680 [3/8081cb7e-e5d5-4fb7-8838-62c6156ec6a4] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:55:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:55:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:55:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:55:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_35 2024-04-24T22:55:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2024-04-24T22:55:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T22:55:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:55:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2024-04-24T22:55:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:55:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_35 2024-04-24T22:55:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T22:55:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:55:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:55:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_35 2024-04-24T22:55:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T22:55:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2024-04-24T22:55:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2024-04-24T22:55:37.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:37.491+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2aa1cd00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:55:37.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:37.491+0000 7fecffbaa700 -1 rbd::mirror::ImageReplayer: 0x557d291a5900 [3/77b74900-6e9c-4b05-b994-e2ce65e571fa] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:55:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:55:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:55:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:55:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_36 2024-04-24T22:55:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2024-04-24T22:55:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T22:55:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:55:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2024-04-24T22:55:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:55:40.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_36 2024-04-24T22:55:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T22:55:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:55:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:55:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_36 2024-04-24T22:55:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T22:55:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2024-04-24T22:55:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2024-04-24T22:55:40.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:40.384+0000 7fecf939d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2aa31d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:55:40.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:40.384+0000 7fecff3a9700 -1 rbd::mirror::ImageReplayer: 0x557d27914f00 [3/d616d270-b42d-464e-90d2-860a74167c56] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:55:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:55:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:55:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:55:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_37 2024-04-24T22:55:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2024-04-24T22:55:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T22:55:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:55:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2024-04-24T22:55:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:55:43.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_37 2024-04-24T22:55:43.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:43.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:43.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T22:55:43.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:55:43.256 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:55:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_37 2024-04-24T22:55:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T22:55:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2024-04-24T22:55:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2024-04-24T22:55:43.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:43.404+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d291d4340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:55:43.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:43.404+0000 7fecffbaa700 -1 rbd::mirror::ImageReplayer: 0x557d291a4c80 [3/a3bbbaa2-a609-401b-8231-3adcfdc73de8] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:55:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:55:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:55:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:55:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_38 2024-04-24T22:55:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2024-04-24T22:55:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T22:55:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:55:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2024-04-24T22:55:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:55:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_38 2024-04-24T22:55:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T22:55:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:55:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:55:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_38 2024-04-24T22:55:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T22:55:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2024-04-24T22:55:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2024-04-24T22:55:46.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:46.416+0000 7fecf939d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2aa32000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:55:46.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:46.416+0000 7fecff3a9700 -1 rbd::mirror::ImageReplayer: 0x557d291a4f00 [3/0b0a4d2e-042e-4de3-97db-78d53010573b] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:55:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:55:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:55:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:55:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_39 2024-04-24T22:55:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2024-04-24T22:55:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T22:55:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:55:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2024-04-24T22:55:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:55:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_39 2024-04-24T22:55:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T22:55:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:55:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:55:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_39 2024-04-24T22:55:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T22:55:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2024-04-24T22:55:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2024-04-24T22:55:49.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:49.413+0000 7fecf939d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2aa32340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:55:49.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:49.413+0000 7fecff3a9700 -1 rbd::mirror::ImageReplayer: 0x557d27915180 [3/74beb076-4d18-4365-b1a8-d0b50a2536f5] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:55:49.685 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-04-24T22:55:49.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-04-24T22:55:49.686 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-04-24T22:55:49.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T22:55:49.685+0000 7ff97d58d700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 71414) UID: 1000 2024-04-24T22:55:49.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:55:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:55:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:55:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:55:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_40 2024-04-24T22:55:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2024-04-24T22:55:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T22:55:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:55:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2024-04-24T22:55:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:55:52.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_40 2024-04-24T22:55:52.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:52.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:52.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T22:55:52.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:55:52.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:55:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_40 2024-04-24T22:55:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T22:55:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2024-04-24T22:55:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2024-04-24T22:55:52.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:52.444+0000 7fecf939d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2b4941a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:55:52.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:52.444+0000 7fecffbaa700 -1 rbd::mirror::ImageReplayer: 0x557d291a5b80 [3/b9d9d1d6-592b-471e-8736-169dcce0a795] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:55:54.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:55:54.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:55:54.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:55:54.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_41 2024-04-24T22:55:54.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2024-04-24T22:55:54.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:54.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:54.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T22:55:54.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:55:54.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2024-04-24T22:55:54.687 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-04-24T22:55:54.687 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:55:54.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:55:54.687 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-04-24T22:55:54.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-24T22:55:54.687 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-24T22:55:54.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-24T22:55:54.690 INFO:tasks.rbd_mirror_thrash:waiting for 112 secs before thrashing 2024-04-24T22:55:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:55:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_41 2024-04-24T22:55:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T22:55:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:55:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:55:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_41 2024-04-24T22:55:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T22:55:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2024-04-24T22:55:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2024-04-24T22:55:55.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:55.444+0000 7fecf8b9c700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557d2aa31380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:55:55.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:55.444+0000 7fecff3a9700 -1 rbd::mirror::ImageReplayer: 0x557d2b055400 [3/f62d1554-3ea2-4d48-bd05-d49435911c52] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:55:55.550 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6948361267960124) > weight (0.25) 2024-04-24T22:55:55.550 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-04-24T22:55:55.550 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-04-24T22:55:55.551 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-04-24T22:55:55.551 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:55:55.550+0000 7fed003ab700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 51331) UID: 1000 2024-04-24T22:55:55.752 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:55:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:55:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:55:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:55:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_42 2024-04-24T22:55:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2024-04-24T22:55:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T22:55:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:55:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2024-04-24T22:55:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:55:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_42 2024-04-24T22:55:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T22:55:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:55:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:55:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_42 2024-04-24T22:55:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:55:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:55:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T22:55:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2024-04-24T22:55:58.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2024-04-24T22:55:58.551 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-04-24T22:55:58.551 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:55:58.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:55:58.552 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-04-24T22:55:58.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-04-24T22:55:58.552 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-04-24T22:55:58.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-04-24T22:55:58.556 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2024-04-24T22:55:59.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:55:59.610+0000 7f9902323700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e8149f4680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:55:59.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:55:59.610+0000 7f9908b30700 -1 rbd::mirror::ImageReplayer: 0x55e814b80c80 [3/abffceee-6d69-43b8-8792-bb1ec5129a1f] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:56:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:56:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_43 2024-04-24T22:56:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2024-04-24T22:56:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T22:56:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:56:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2024-04-24T22:56:01.307 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:56:01.333 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_43 2024-04-24T22:56:01.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:01.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:01.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T22:56:01.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:56:01.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:56:01.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_43 2024-04-24T22:56:01.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:01.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:01.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T22:56:01.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2024-04-24T22:56:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2024-04-24T22:56:01.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:56:01.477+0000 7f9902323700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e814a8dba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:56:01.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:56:01.477+0000 7f9908b30700 -1 rbd::mirror::ImageReplayer: 0x55e814b80780 [3/50cbbefd-55ee-46b7-9b9e-c53db4b8ed79] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:56:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:56:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_44 2024-04-24T22:56:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2024-04-24T22:56:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T22:56:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:56:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2024-04-24T22:56:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:56:04.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_44 2024-04-24T22:56:04.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:04.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:04.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T22:56:04.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:56:04.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:56:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_44 2024-04-24T22:56:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T22:56:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2024-04-24T22:56:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2024-04-24T22:56:04.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:56:04.498+0000 7f9901b22700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e81237a680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:56:04.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:56:04.498+0000 7f990832f700 -1 rbd::mirror::ImageReplayer: 0x55e814b80000 [3/1cecaa75-0bd6-483a-abc1-814cd6c8b99b] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:56:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:56:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_45 2024-04-24T22:56:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2024-04-24T22:56:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T22:56:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:56:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2024-04-24T22:56:07.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:56:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_45 2024-04-24T22:56:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T22:56:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:56:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:56:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_45 2024-04-24T22:56:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T22:56:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2024-04-24T22:56:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2024-04-24T22:56:07.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:56:07.502+0000 7f9902323700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e8149f56c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:56:07.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:56:07.502+0000 7f9908b30700 -1 rbd::mirror::ImageReplayer: 0x55e814b80280 [3/94475b78-a054-4f7b-8225-9f06e6be39b9] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:56:09.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:56:09.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:09.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:09.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_46 2024-04-24T22:56:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2024-04-24T22:56:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T22:56:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:56:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2024-04-24T22:56:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:56:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_46 2024-04-24T22:56:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T22:56:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:56:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:56:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_46 2024-04-24T22:56:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T22:56:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2024-04-24T22:56:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2024-04-24T22:56:10.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:56:10.394+0000 7f9902323700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e814b0bd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:56:10.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:56:10.394+0000 7f9908b30700 -1 rbd::mirror::ImageReplayer: 0x55e814b80a00 [3/aaca0d68-80e6-4367-94cf-716faf3d8e67] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:56:12.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:56:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_47 2024-04-24T22:56:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2024-04-24T22:56:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T22:56:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:56:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2024-04-24T22:56:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:56:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_47 2024-04-24T22:56:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T22:56:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:56:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:56:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_47 2024-04-24T22:56:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T22:56:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2024-04-24T22:56:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2024-04-24T22:56:13.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:56:13.413+0000 7f9902323700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e8149a4b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:56:13.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:56:13.413+0000 7f990832f700 -1 rbd::mirror::ImageReplayer: 0x55e814b81400 [3/b943dd73-ded7-434d-8abb-ad7480804db1] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:56:15.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:56:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_48 2024-04-24T22:56:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2024-04-24T22:56:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T22:56:15.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:56:15.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2024-04-24T22:56:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:56:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_48 2024-04-24T22:56:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T22:56:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:56:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:56:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_48 2024-04-24T22:56:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T22:56:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2024-04-24T22:56:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2024-04-24T22:56:16.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:56:16.437+0000 7f9901b22700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e8149a4340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:56:16.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:56:16.437+0000 7f990832f700 -1 rbd::mirror::ImageReplayer: 0x55e814b80500 [3/4b92c784-d7e2-461f-9fc8-08abc3ca10c8] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:56:18.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:56:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_49 2024-04-24T22:56:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2024-04-24T22:56:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T22:56:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:56:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2024-04-24T22:56:19.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:56:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_49 2024-04-24T22:56:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T22:56:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:56:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:56:19.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_49 2024-04-24T22:56:19.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T22:56:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2024-04-24T22:56:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2024-04-24T22:56:19.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:56:19.445+0000 7f9902323700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e8149d6340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:56:19.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:56:19.445+0000 7f990832f700 -1 rbd::mirror::ImageReplayer: 0x55e814b80f00 [3/ae301ebb-cfbe-4565-b97c-7f626fd5375e] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:56:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:56:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_50 2024-04-24T22:56:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2024-04-24T22:56:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T22:56:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local snap=snap 2024-04-24T22:56:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2024-04-24T22:56:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing snap: 100% complete...done. 2024-04-24T22:56:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_50 2024-04-24T22:56:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T22:56:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:56:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:56:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_50 2024-04-24T22:56:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T22:56:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2024-04-24T22:56:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2024-04-24T22:56:22.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:56:22.460+0000 7f9901b22700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e8149a4340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:56:22.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:56:22.460+0000 7f990832f700 -1 rbd::mirror::ImageReplayer: 0x55e8116b9b80 [3/19c08b8e-0c4b-4471-adf6-b6a40dd52550] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:56:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-04-24T22:56:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ testlog 'TEST: image deletions should propagate' 2024-04-24T22:56:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ tee -a /tmp/tmp.DCvHjwkiaM/rbd-mirror.test.log 2024-04-24T22:56:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ date '+%F %T' 2024-04-24T22:56:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo 2024-04-24 22:56:24 TEST: image deletions should propagate 2024-04-24T22:56:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:2024-04-24 22:56:24 TEST: image deletions should propagate 2024-04-24T22:56:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_pool_images cluster1 mirror 0 2024-04-24T22:56:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_count=0 2024-04-24T22:56:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local count 2024-04-24T22:56:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local last_count=0 2024-04-24T22:56:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ true 2024-04-24T22:56:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 40 2024-04-24T22:56:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in `seq 1 40` 2024-04-24T22:56:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 1 -ne 1 2024-04-24T22:56:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-04-24T22:56:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep 'images: ' 2024-04-24T22:56:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut '-d ' -f 2 2024-04-24T22:56:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: failed to query services: (13) Permission denied 2024-04-24T22:56:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ count=0 2024-04-24T22:56:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 0 = 0 2024-04-24T22:56:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2024-04-24T22:56:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state 2024-04-24T22:56:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 40 2024-04-24T22:56:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in `seq 1 40` 2024-04-24T22:56:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 1 -ne 1 2024-04-24T22:56:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-04-24T22:56:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep 'image health:' 2024-04-24T22:56:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cut '-d ' -f 3 2024-04-24T22:56:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: failed to query services: (13) Permission denied 2024-04-24T22:56:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ state=OK 2024-04-24T22:56:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test OK = ERROR 2024-04-24T22:56:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test OK = OK 2024-04-24T22:56:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 50 2024-04-24T22:56:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_1 2024-04-24T22:56:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-04-24T22:56:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:56:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_1 2024-04-24T22:56:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T22:56:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-24T22:56:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:24.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-04-24T22:56:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-04-24T22:56:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:56:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_1 2024-04-24T22:56:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T22:56:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-24T22:56:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-04-24T22:56:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_2 2024-04-24T22:56:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-04-24T22:56:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:56:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_2 2024-04-24T22:56:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_2 2024-04-24T22:56:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-24T22:56:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-04-24T22:56:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-04-24T22:56:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:56:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_2 2024-04-24T22:56:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_2 2024-04-24T22:56:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-24T22:56:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-04-24T22:56:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_3 2024-04-24T22:56:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-04-24T22:56:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:56:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_3 2024-04-24T22:56:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T22:56:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-24T22:56:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-04-24T22:56:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-04-24T22:56:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:56:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_3 2024-04-24T22:56:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T22:56:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-24T22:56:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-04-24T22:56:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_4 2024-04-24T22:56:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-04-24T22:56:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:56:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_4 2024-04-24T22:56:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T22:56:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-24T22:56:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:25.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-04-24T22:56:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-04-24T22:56:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:56:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_4 2024-04-24T22:56:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T22:56:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-24T22:56:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-04-24T22:56:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_5 2024-04-24T22:56:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-04-24T22:56:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:56:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_5 2024-04-24T22:56:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_5 2024-04-24T22:56:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-24T22:56:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:25.600 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-04-24T22:56:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-04-24T22:56:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:56:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_5 2024-04-24T22:56:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_5 2024-04-24T22:56:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-24T22:56:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-04-24T22:56:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_6 2024-04-24T22:56:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-04-24T22:56:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T22:56:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_6 2024-04-24T22:56:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T22:56:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-24T22:56:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-04-24T22:56:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-04-24T22:56:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T22:56:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_6 2024-04-24T22:56:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T22:56:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-24T22:56:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-04-24T22:56:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_7 2024-04-24T22:56:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-04-24T22:56:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T22:56:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_7 2024-04-24T22:56:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T22:56:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-24T22:56:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:25.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-04-24T22:56:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-04-24T22:56:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T22:56:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_7 2024-04-24T22:56:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T22:56:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-24T22:56:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-04-24T22:56:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_8 2024-04-24T22:56:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-04-24T22:56:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T22:56:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_8 2024-04-24T22:56:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T22:56:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-24T22:56:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:26.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-04-24T22:56:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-04-24T22:56:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T22:56:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:26.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_8 2024-04-24T22:56:26.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:26.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:26.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T22:56:26.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-24T22:56:26.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-04-24T22:56:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_9 2024-04-24T22:56:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-04-24T22:56:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T22:56:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_9 2024-04-24T22:56:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_9 2024-04-24T22:56:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-24T22:56:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-04-24T22:56:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-04-24T22:56:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T22:56:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_9 2024-04-24T22:56:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_9 2024-04-24T22:56:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-24T22:56:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:26.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-04-24T22:56:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_10 2024-04-24T22:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-04-24T22:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T22:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_10 2024-04-24T22:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T22:56:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-24T22:56:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-04-24T22:56:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-04-24T22:56:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T22:56:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_10 2024-04-24T22:56:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T22:56:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-24T22:56:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-04-24T22:56:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_11 2024-04-24T22:56:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2024-04-24T22:56:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T22:56:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_11 2024-04-24T22:56:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T22:56:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-04-24T22:56:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-04-24T22:56:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2024-04-24T22:56:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_11 2024-04-24T22:56:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_11 2024-04-24T22:56:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_11 2024-04-24T22:56:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-04-24T22:56:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-04-24T22:56:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_12 2024-04-24T22:56:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2024-04-24T22:56:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T22:56:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_12 2024-04-24T22:56:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T22:56:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-04-24T22:56:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-04-24T22:56:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2024-04-24T22:56:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_12 2024-04-24T22:56:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:26.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_12 2024-04-24T22:56:26.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:26.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:26.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_12 2024-04-24T22:56:26.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-04-24T22:56:26.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-04-24T22:56:26.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:26.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:26.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:26.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:26.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:26.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_13 2024-04-24T22:56:26.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2024-04-24T22:56:26.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:26.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:26.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T22:56:26.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:26.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:26.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:26.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:26.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_13 2024-04-24T22:56:26.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:26.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:26.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_13 2024-04-24T22:56:26.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-04-24T22:56:26.884 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-04-24T22:56:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2024-04-24T22:56:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_13 2024-04-24T22:56:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_13 2024-04-24T22:56:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_13 2024-04-24T22:56:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-04-24T22:56:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-04-24T22:56:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_14 2024-04-24T22:56:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2024-04-24T22:56:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T22:56:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_14 2024-04-24T22:56:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_14 2024-04-24T22:56:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-04-24T22:56:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-04-24T22:56:27.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:27.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:27.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2024-04-24T22:56:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_14 2024-04-24T22:56:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_14 2024-04-24T22:56:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_14 2024-04-24T22:56:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-04-24T22:56:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-04-24T22:56:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_15 2024-04-24T22:56:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2024-04-24T22:56:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T22:56:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_15 2024-04-24T22:56:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T22:56:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-04-24T22:56:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:27.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-04-24T22:56:27.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:27.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:27.240 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2024-04-24T22:56:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_15 2024-04-24T22:56:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_15 2024-04-24T22:56:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_15 2024-04-24T22:56:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-04-24T22:56:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-04-24T22:56:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_16 2024-04-24T22:56:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2024-04-24T22:56:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T22:56:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_16 2024-04-24T22:56:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_16 2024-04-24T22:56:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-04-24T22:56:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-04-24T22:56:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2024-04-24T22:56:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_16 2024-04-24T22:56:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:27.505 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_16 2024-04-24T22:56:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_16 2024-04-24T22:56:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-04-24T22:56:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-04-24T22:56:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_17 2024-04-24T22:56:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2024-04-24T22:56:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T22:56:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_17 2024-04-24T22:56:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_17 2024-04-24T22:56:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-04-24T22:56:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-04-24T22:56:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2024-04-24T22:56:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_17 2024-04-24T22:56:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_17 2024-04-24T22:56:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_17 2024-04-24T22:56:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-04-24T22:56:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-04-24T22:56:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_18 2024-04-24T22:56:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2024-04-24T22:56:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T22:56:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_18 2024-04-24T22:56:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_18 2024-04-24T22:56:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-04-24T22:56:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-04-24T22:56:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2024-04-24T22:56:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_18 2024-04-24T22:56:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_18 2024-04-24T22:56:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_18 2024-04-24T22:56:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-04-24T22:56:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-04-24T22:56:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_19 2024-04-24T22:56:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2024-04-24T22:56:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T22:56:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_19 2024-04-24T22:56:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_19 2024-04-24T22:56:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-04-24T22:56:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-04-24T22:56:27.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:27.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:27.896 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2024-04-24T22:56:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_19 2024-04-24T22:56:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_19 2024-04-24T22:56:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_19 2024-04-24T22:56:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-04-24T22:56:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-04-24T22:56:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_20 2024-04-24T22:56:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2024-04-24T22:56:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T22:56:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_20 2024-04-24T22:56:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T22:56:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-04-24T22:56:28.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-04-24T22:56:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2024-04-24T22:56:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_20 2024-04-24T22:56:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_20 2024-04-24T22:56:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_20 2024-04-24T22:56:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-04-24T22:56:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-04-24T22:56:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_21 2024-04-24T22:56:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2024-04-24T22:56:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T22:56:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_21 2024-04-24T22:56:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T22:56:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-04-24T22:56:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-04-24T22:56:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2024-04-24T22:56:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_21 2024-04-24T22:56:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_21 2024-04-24T22:56:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:28.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_21 2024-04-24T22:56:28.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-04-24T22:56:28.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-04-24T22:56:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_22 2024-04-24T22:56:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2024-04-24T22:56:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T22:56:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_22 2024-04-24T22:56:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T22:56:28.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-04-24T22:56:28.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:28.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-04-24T22:56:28.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:28.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:28.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2024-04-24T22:56:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_22 2024-04-24T22:56:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_22 2024-04-24T22:56:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_22 2024-04-24T22:56:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-04-24T22:56:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-04-24T22:56:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_23 2024-04-24T22:56:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2024-04-24T22:56:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T22:56:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_23 2024-04-24T22:56:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T22:56:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-04-24T22:56:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:28.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-04-24T22:56:28.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:28.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:28.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:28.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2024-04-24T22:56:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_23 2024-04-24T22:56:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_23 2024-04-24T22:56:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_23 2024-04-24T22:56:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-04-24T22:56:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-04-24T22:56:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_24 2024-04-24T22:56:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2024-04-24T22:56:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T22:56:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_24 2024-04-24T22:56:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_24 2024-04-24T22:56:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-04-24T22:56:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:28.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-04-24T22:56:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2024-04-24T22:56:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_24 2024-04-24T22:56:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_24 2024-04-24T22:56:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_24 2024-04-24T22:56:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-04-24T22:56:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-04-24T22:56:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_25 2024-04-24T22:56:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2024-04-24T22:56:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T22:56:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_25 2024-04-24T22:56:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T22:56:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-04-24T22:56:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:28.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-04-24T22:56:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2024-04-24T22:56:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_25 2024-04-24T22:56:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_25 2024-04-24T22:56:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_25 2024-04-24T22:56:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-04-24T22:56:28.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-04-24T22:56:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_26 2024-04-24T22:56:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2024-04-24T22:56:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T22:56:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_26 2024-04-24T22:56:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_26 2024-04-24T22:56:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-04-24T22:56:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-04-24T22:56:29.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:29.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:29.052 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2024-04-24T22:56:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_26 2024-04-24T22:56:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:29.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:29.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_26 2024-04-24T22:56:29.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:29.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:29.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_26 2024-04-24T22:56:29.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-04-24T22:56:29.156 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-04-24T22:56:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_27 2024-04-24T22:56:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2024-04-24T22:56:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T22:56:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_27 2024-04-24T22:56:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T22:56:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-04-24T22:56:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-04-24T22:56:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2024-04-24T22:56:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_27 2024-04-24T22:56:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_27 2024-04-24T22:56:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_27 2024-04-24T22:56:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-04-24T22:56:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-04-24T22:56:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_28 2024-04-24T22:56:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2024-04-24T22:56:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T22:56:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_28 2024-04-24T22:56:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_28 2024-04-24T22:56:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-04-24T22:56:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-04-24T22:56:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2024-04-24T22:56:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_28 2024-04-24T22:56:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_28 2024-04-24T22:56:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_28 2024-04-24T22:56:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-04-24T22:56:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:29.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-04-24T22:56:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_29 2024-04-24T22:56:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2024-04-24T22:56:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T22:56:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_29 2024-04-24T22:56:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_29 2024-04-24T22:56:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-04-24T22:56:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-04-24T22:56:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2024-04-24T22:56:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_29 2024-04-24T22:56:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_29 2024-04-24T22:56:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_29 2024-04-24T22:56:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-04-24T22:56:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-04-24T22:56:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_30 2024-04-24T22:56:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2024-04-24T22:56:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T22:56:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_30 2024-04-24T22:56:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_30 2024-04-24T22:56:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-04-24T22:56:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-04-24T22:56:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2024-04-24T22:56:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_30 2024-04-24T22:56:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_30 2024-04-24T22:56:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_30 2024-04-24T22:56:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-04-24T22:56:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:29.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-04-24T22:56:29.842 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:29.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:29.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:29.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:29.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:29.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_31 2024-04-24T22:56:29.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2024-04-24T22:56:29.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:29.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:29.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T22:56:29.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:29.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:29.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:29.843 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_31 2024-04-24T22:56:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T22:56:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-04-24T22:56:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:29.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-04-24T22:56:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2024-04-24T22:56:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_31 2024-04-24T22:56:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_31 2024-04-24T22:56:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_31 2024-04-24T22:56:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-04-24T22:56:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-04-24T22:56:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_32 2024-04-24T22:56:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2024-04-24T22:56:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T22:56:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_32 2024-04-24T22:56:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T22:56:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-04-24T22:56:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-04-24T22:56:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2024-04-24T22:56:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_32 2024-04-24T22:56:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_32 2024-04-24T22:56:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_32 2024-04-24T22:56:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-04-24T22:56:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:30.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-04-24T22:56:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_33 2024-04-24T22:56:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2024-04-24T22:56:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T22:56:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_33 2024-04-24T22:56:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T22:56:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-04-24T22:56:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-04-24T22:56:30.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:30.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:30.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2024-04-24T22:56:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_33 2024-04-24T22:56:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_33 2024-04-24T22:56:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_33 2024-04-24T22:56:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-04-24T22:56:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-04-24T22:56:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_34 2024-04-24T22:56:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2024-04-24T22:56:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T22:56:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_34 2024-04-24T22:56:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T22:56:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-04-24T22:56:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-04-24T22:56:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2024-04-24T22:56:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_34 2024-04-24T22:56:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_34 2024-04-24T22:56:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_34 2024-04-24T22:56:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-04-24T22:56:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-04-24T22:56:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_35 2024-04-24T22:56:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2024-04-24T22:56:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T22:56:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_35 2024-04-24T22:56:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T22:56:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-04-24T22:56:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-04-24T22:56:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2024-04-24T22:56:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_35 2024-04-24T22:56:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_35 2024-04-24T22:56:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_35 2024-04-24T22:56:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-04-24T22:56:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-04-24T22:56:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_36 2024-04-24T22:56:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2024-04-24T22:56:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T22:56:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_36 2024-04-24T22:56:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_36 2024-04-24T22:56:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-04-24T22:56:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:30.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-04-24T22:56:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2024-04-24T22:56:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_36 2024-04-24T22:56:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_36 2024-04-24T22:56:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_36 2024-04-24T22:56:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-04-24T22:56:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-04-24T22:56:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_37 2024-04-24T22:56:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2024-04-24T22:56:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T22:56:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_37 2024-04-24T22:56:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_37 2024-04-24T22:56:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-04-24T22:56:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-04-24T22:56:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2024-04-24T22:56:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_37 2024-04-24T22:56:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_37 2024-04-24T22:56:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_37 2024-04-24T22:56:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-04-24T22:56:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-04-24T22:56:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_38 2024-04-24T22:56:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2024-04-24T22:56:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T22:56:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_38 2024-04-24T22:56:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_38 2024-04-24T22:56:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-04-24T22:56:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-04-24T22:56:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2024-04-24T22:56:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_38 2024-04-24T22:56:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_38 2024-04-24T22:56:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_38 2024-04-24T22:56:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-04-24T22:56:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-04-24T22:56:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_39 2024-04-24T22:56:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2024-04-24T22:56:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T22:56:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_39 2024-04-24T22:56:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_39 2024-04-24T22:56:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-04-24T22:56:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-04-24T22:56:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2024-04-24T22:56:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_39 2024-04-24T22:56:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_39 2024-04-24T22:56:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_39 2024-04-24T22:56:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-04-24T22:56:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-04-24T22:56:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_40 2024-04-24T22:56:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2024-04-24T22:56:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T22:56:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_40 2024-04-24T22:56:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T22:56:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-04-24T22:56:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-04-24T22:56:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2024-04-24T22:56:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_40 2024-04-24T22:56:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_40 2024-04-24T22:56:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_40 2024-04-24T22:56:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-04-24T22:56:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-04-24T22:56:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_41 2024-04-24T22:56:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2024-04-24T22:56:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T22:56:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_41 2024-04-24T22:56:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T22:56:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-04-24T22:56:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-04-24T22:56:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2024-04-24T22:56:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_41 2024-04-24T22:56:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_41 2024-04-24T22:56:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_41 2024-04-24T22:56:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-04-24T22:56:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-04-24T22:56:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_42 2024-04-24T22:56:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2024-04-24T22:56:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T22:56:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_42 2024-04-24T22:56:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_42 2024-04-24T22:56:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-04-24T22:56:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-04-24T22:56:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2024-04-24T22:56:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_42 2024-04-24T22:56:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_42 2024-04-24T22:56:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_42 2024-04-24T22:56:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-04-24T22:56:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-04-24T22:56:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_43 2024-04-24T22:56:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2024-04-24T22:56:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T22:56:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_43 2024-04-24T22:56:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T22:56:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-04-24T22:56:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-04-24T22:56:31.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:31.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:31.839 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2024-04-24T22:56:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_43 2024-04-24T22:56:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_43 2024-04-24T22:56:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_43 2024-04-24T22:56:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-04-24T22:56:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-04-24T22:56:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:31.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_44 2024-04-24T22:56:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2024-04-24T22:56:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T22:56:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_44 2024-04-24T22:56:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T22:56:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-04-24T22:56:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-04-24T22:56:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:32.105 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2024-04-24T22:56:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_44 2024-04-24T22:56:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_44 2024-04-24T22:56:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_44 2024-04-24T22:56:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-04-24T22:56:32.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-04-24T22:56:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_45 2024-04-24T22:56:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2024-04-24T22:56:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T22:56:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_45 2024-04-24T22:56:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T22:56:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-04-24T22:56:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-04-24T22:56:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:32.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2024-04-24T22:56:32.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:32.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:32.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_45 2024-04-24T22:56:32.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_45 2024-04-24T22:56:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_45 2024-04-24T22:56:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-04-24T22:56:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:32.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-04-24T22:56:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_46 2024-04-24T22:56:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2024-04-24T22:56:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T22:56:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_46 2024-04-24T22:56:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T22:56:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-04-24T22:56:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-04-24T22:56:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2024-04-24T22:56:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_46 2024-04-24T22:56:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_46 2024-04-24T22:56:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_46 2024-04-24T22:56:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-04-24T22:56:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-04-24T22:56:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_47 2024-04-24T22:56:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2024-04-24T22:56:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T22:56:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_47 2024-04-24T22:56:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T22:56:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-04-24T22:56:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-04-24T22:56:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2024-04-24T22:56:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_47 2024-04-24T22:56:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_47 2024-04-24T22:56:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_47 2024-04-24T22:56:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-04-24T22:56:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:32.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-04-24T22:56:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_48 2024-04-24T22:56:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2024-04-24T22:56:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T22:56:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:32.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:32.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_48 2024-04-24T22:56:32.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:32.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:32.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T22:56:32.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-04-24T22:56:32.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-04-24T22:56:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2024-04-24T22:56:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_48 2024-04-24T22:56:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_48 2024-04-24T22:56:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_48 2024-04-24T22:56:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-04-24T22:56:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-04-24T22:56:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_49 2024-04-24T22:56:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2024-04-24T22:56:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T22:56:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_49 2024-04-24T22:56:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T22:56:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-04-24T22:56:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:32.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-04-24T22:56:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2024-04-24T22:56:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_49 2024-04-24T22:56:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_49 2024-04-24T22:56:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_49 2024-04-24T22:56:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-04-24T22:56:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-04-24T22:56:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-04-24T22:56:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_50 2024-04-24T22:56:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2024-04-24T22:56:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T22:56:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:56:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_50 2024-04-24T22:56:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T22:56:32.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-04-24T22:56:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-04-24T22:56:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:56:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2024-04-24T22:56:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_50 2024-04-24T22:56:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:56:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_50 2024-04-24T22:56:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_50 2024-04-24T22:56:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-04-24T22:56:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-04-24T22:56:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:56:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:56:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ testlog 'TEST: delete images during bootstrap' 2024-04-24T22:56:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ tee -a /tmp/tmp.DCvHjwkiaM/rbd-mirror.test.log 2024-04-24T22:56:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ date '+%F %T' 2024-04-24T22:56:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo 2024-04-24 22:56:33 TEST: delete images during bootstrap 2024-04-24T22:56:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:2024-04-24 22:56:33 TEST: delete images during bootstrap 2024-04-24T22:56:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_pool_mirror_mode cluster1 mirror image 2024-04-24T22:56:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=image 2024-04-24T22:56:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2024-04-24T22:56:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd: mirroring is already configured for image mode 2024-04-24T22:56:33.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-04-24T22:56:33.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:33.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=image 2024-04-24T22:56:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-04-24T22:56:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd: mirroring is already configured for image mode 2024-04-24T22:56:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ start_mirror cluster1 2024-04-24T22:56:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance 2024-04-24T22:56:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set_cluster_instance cluster1 cluster instance 2024-04-24T22:56:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local val=cluster1 2024-04-24T22:56:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster_var_name=cluster 2024-04-24T22:56:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local instance_var_name=instance 2024-04-24T22:56:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cluster=cluster1 2024-04-24T22:56:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=cluster1 2024-04-24T22:56:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' cluster1 = cluster1 ']' 2024-04-24T22:56:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ instance=0 2024-04-24T22:56:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval cluster=cluster1 2024-04-24T22:56:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ cluster=cluster1 2024-04-24T22:56:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ eval instance=0 2024-04-24T22:56:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ instance=0 2024-04-24T22:56:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T22:56:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T22:56:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=test 2024-04-24T22:56:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 1 10 2024-04-24T22:56:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:56:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_1 2024-04-24T22:56:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_1 512M 2024-04-24T22:56:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:56:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:56:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:56:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:56:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:56:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 512M ']' 2024-04-24T22:56:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=512M 2024-04-24T22:56:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:56:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2024-04-24T22:56:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_1 2024-04-24T22:56:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:56:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:56:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-04-24T22:56:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:56:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-04-24T22:56:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_1': 2024-04-24T22:56:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T22:56:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:56:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:56:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 19588e69fa42 2024-04-24T22:56:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.19588e69fa42 2024-04-24T22:56:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:56:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:56:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:56:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:56:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:56:33 2024 2024-04-24T22:56:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:56:33 2024 2024-04-24T22:56:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:56:33 2024 2024-04-24T22:56:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:56:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:56:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: a0d792d5-b2a9-4d85-9b1e-6da576622d8f 2024-04-24T22:56:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:56:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_1 2024-04-24T22:56:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:56:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:56:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=13 2024-04-24T22:56:34.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:56:34.368 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:56:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 11092992 bytes 2024-04-24T22:56:34.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 102429696 bytes 2024-04-24T22:56:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 126074880 bytes 2024-04-24T22:56:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 145751040 bytes 2024-04-24T22:56:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 177893376 bytes 2024-04-24T22:56:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 227173376 bytes 2024-04-24T22:56:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 253562880 bytes 2024-04-24T22:56:35.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 270776320 bytes 2024-04-24T22:56:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 294889472 bytes 2024-04-24T22:56:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 426215424 bytes 2024-04-24T22:56:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 485572608 bytes 2024-04-24T22:56:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 527909888 bytes 2024-04-24T22:56:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 601612288 bytes 2024-04-24T22:56:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 624286720 bytes 2024-04-24T22:56:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 671598592 bytes 2024-04-24T22:56:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 739454976 bytes 2024-04-24T22:56:36.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 742594560 bytes 2024-04-24T22:56:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 830643200 bytes 2024-04-24T22:56:37.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 865628160 bytes 2024-04-24T22:56:37.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 866252800 bytes 2024-04-24T22:56:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 890083328 bytes 2024-04-24T22:56:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 949948416 bytes 2024-04-24T22:56:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 983908352 bytes 2024-04-24T22:56:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1056872448 bytes 2024-04-24T22:56:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1120051200 bytes 2024-04-24T22:56:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1127621632 bytes 2024-04-24T22:56:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1183902720 bytes 2024-04-24T22:56:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1230679040 bytes 2024-04-24T22:56:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1265212416 bytes 2024-04-24T22:56:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1282347008 bytes 2024-04-24T22:56:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1314181120 bytes 2024-04-24T22:56:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1379788800 bytes 2024-04-24T22:56:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1428815872 bytes 2024-04-24T22:56:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1445913600 bytes 2024-04-24T22:56:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1457336320 bytes 2024-04-24T22:56:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1542196224 bytes 2024-04-24T22:56:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1607051264 bytes 2024-04-24T22:56:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1632851968 bytes 2024-04-24T22:56:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1663847424 bytes 2024-04-24T22:56:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1698518016 bytes 2024-04-24T22:56:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1778306048 bytes 2024-04-24T22:56:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1832981504 bytes 2024-04-24T22:56:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1870722048 bytes 2024-04-24T22:56:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1915237376 bytes 2024-04-24T22:56:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 2015102976 bytes 2024-04-24T22:56:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 2030117888 bytes 2024-04-24T22:56:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 2084210688 bytes 2024-04-24T22:56:41.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2128885760 bytes 2024-04-24T22:56:41.530 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2188294144 bytes 2024-04-24T22:56:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2216792064 bytes 2024-04-24T22:56:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2244458496 bytes 2024-04-24T22:56:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2285375488 bytes 2024-04-24T22:56:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2359424000 bytes 2024-04-24T22:56:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2360049664 bytes 2024-04-24T22:56:42.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2374839296 bytes 2024-04-24T22:56:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2419598336 bytes 2024-04-24T22:56:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2500469760 bytes 2024-04-24T22:56:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2533369856 bytes 2024-04-24T22:56:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2598330368 bytes 2024-04-24T22:56:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2643211264 bytes 2024-04-24T22:56:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2694177792 bytes 2024-04-24T22:56:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2737554432 bytes 2024-04-24T22:56:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2745686016 bytes 2024-04-24T22:56:43.695 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2785165312 bytes 2024-04-24T22:56:43.844 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2833592320 bytes 2024-04-24T22:56:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2875084800 bytes 2024-04-24T22:56:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2901833728 bytes 2024-04-24T22:56:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2919081984 bytes 2024-04-24T22:56:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2977378304 bytes 2024-04-24T22:56:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 3056558080 bytes 2024-04-24T22:56:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 3092113408 bytes 2024-04-24T22:56:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 3148940288 bytes 2024-04-24T22:56:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 3187529728 bytes 2024-04-24T22:56:45.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 3247507456 bytes 2024-04-24T22:56:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 3315971072 bytes 2024-04-24T22:56:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 3347603456 bytes 2024-04-24T22:56:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 3381501952 bytes 2024-04-24T22:56:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 3407144960 bytes 2024-04-24T22:56:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7900 IOs, 3460523008 bytes 2024-04-24T22:56:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8000 IOs, 3516956672 bytes 2024-04-24T22:56:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8100 IOs, 3539777536 bytes 2024-04-24T22:56:46.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8200 IOs, 3620476928 bytes 2024-04-24T22:56:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8300 IOs, 3642733568 bytes 2024-04-24T22:56:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8400 IOs, 3699182592 bytes 2024-04-24T22:56:46.859 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8500 IOs, 3765300224 bytes 2024-04-24T22:56:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8600 IOs, 3817313280 bytes 2024-04-24T22:56:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8700 IOs, 3843847168 bytes 2024-04-24T22:56:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8800 IOs, 3919663104 bytes 2024-04-24T22:56:47.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:56:47.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:56:47.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:56:47.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:47.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2024-04-24T22:56:47.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:47.374 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:47.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:56:47.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=present 2024-04-24T22:56:47.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:56:47.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:56:47.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:56:47.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_1 2024-04-24T22:56:47.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:47.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:47.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T22:56:47.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-24T22:56:47.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id=10dc477f37db 2024-04-24T22:56:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:56:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:56:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_1 present 10dc477f37db 2024-04-24T22:56:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:56:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:56:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=present 2024-04-24T22:56:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id=10dc477f37db 2024-04-24T22:56:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:56:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:56:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_1 2024-04-24T22:56:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:56:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:56:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T22:56:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-24T22:56:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:56:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id=10dc477f37db 2024-04-24T22:56:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 10dc477f37db 2024-04-24T22:56:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -z 10dc477f37db -o 10dc477f37db = 10dc477f37db 2024-04-24T22:56:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=present 2024-04-24T22:56:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test present = present 2024-04-24T22:56:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:56:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ disable_mirror cluster2 mirror image_1 2024-04-24T22:56:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:56:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:56:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:56:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2024-04-24T22:56:57.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:56:57.665+0000 7f9902323700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e8149d49c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:56:57.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:56:57.665+0000 7f9908b30700 -1 rbd::mirror::ImageReplayer: 0x55e814b80c80 [3/a0d792d5-b2a9-4d85-9b1e-6da576622d8f] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:57:16.557 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9377710377327458) > weight (0.25) 2024-04-24T22:57:16.557 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-04-24T22:57:16.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-04-24T22:57:16.558 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-04-24T22:57:16.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:57:16.557+0000 7f24d1399700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 77852) UID: 1000 2024-04-24T22:57:16.759 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:57:18.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T22:57:18.394+0000 7f4a5070b700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55dd066e9000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=368 2024-04-24T22:57:18.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T22:57:18.394+0000 7f4a5070b700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55dd066e9000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=368, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=d9b73256-b8ac-4751-9b60-95a0234fa37c, primary_snap_id=170, last_copied_object_number=128, snap_seqs={368=18446744073709551614}] 2024-04-24T22:57:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring disabled 2024-04-24T22:57:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-04-24T22:57:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:57:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:57:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:57:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:57:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:57:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:57:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:57:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_1 2024-04-24T22:57:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:57:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:57:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T22:57:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-24T22:57:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:57:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-04-24T22:57:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:57:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:57:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:57:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-04-24T22:57:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:57:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:57:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:57:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:57:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:57:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:57:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:57:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_1 2024-04-24T22:57:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:57:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:57:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_1 2024-04-24T22:57:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-04-24T22:57:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:57:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-04-24T22:57:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:57:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:57:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:57:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:57:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ purge_snapshots cluster2 mirror image_1 2024-04-24T22:57:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:57:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:57:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:57:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2024-04-24T22:57:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_1 2024-04-24T22:57:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:57:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:57:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:57:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:57:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:57:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_1 2024-04-24T22:57:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:57:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:57:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_1 2024-04-24T22:57:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-04-24T22:57:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-04-24T22:57:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-24T22:57:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:57:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:57:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_2 2024-04-24T22:57:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_2 512M 2024-04-24T22:57:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:57:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:57:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:57:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:57:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:57:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:57:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:57:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 512M ']' 2024-04-24T22:57:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=512M 2024-04-24T22:57:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:57:18.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2024-04-24T22:57:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_2 2024-04-24T22:57:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:57:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:57:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:57:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:57:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-04-24T22:57:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:57:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-04-24T22:57:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_2': 2024-04-24T22:57:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T22:57:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:57:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:57:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 19707a069233 2024-04-24T22:57:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.19707a069233 2024-04-24T22:57:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:57:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:57:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:57:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:57:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:57:18 2024 2024-04-24T22:57:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:57:18 2024 2024-04-24T22:57:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:57:18 2024 2024-04-24T22:57:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:57:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:57:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 175ad86a-8c60-47a3-835f-e512e065658b 2024-04-24T22:57:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:57:19.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_2 2024-04-24T22:57:19.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:57:19.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:57:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:57:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:57:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=9 2024-04-24T22:57:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:57:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:57:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 60248064 bytes 2024-04-24T22:57:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 94837760 bytes 2024-04-24T22:57:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 171541504 bytes 2024-04-24T22:57:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 238643200 bytes 2024-04-24T22:57:20.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 289811456 bytes 2024-04-24T22:57:20.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 395234304 bytes 2024-04-24T22:57:20.739 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 441966592 bytes 2024-04-24T22:57:20.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 491119616 bytes 2024-04-24T22:57:21.012 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 522118144 bytes 2024-04-24T22:57:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 558320640 bytes 2024-04-24T22:57:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 590926848 bytes 2024-04-24T22:57:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 645881856 bytes 2024-04-24T22:57:21.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 738683904 bytes 2024-04-24T22:57:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 760507392 bytes 2024-04-24T22:57:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 805344256 bytes 2024-04-24T22:57:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 841792512 bytes 2024-04-24T22:57:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 893714432 bytes 2024-04-24T22:57:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 912047104 bytes 2024-04-24T22:57:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 962644992 bytes 2024-04-24T22:57:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 999013376 bytes 2024-04-24T22:57:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 1032741888 bytes 2024-04-24T22:57:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 1073725440 bytes 2024-04-24T22:57:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1120102400 bytes 2024-04-24T22:57:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1196512256 bytes 2024-04-24T22:57:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1271236608 bytes 2024-04-24T22:57:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1301915648 bytes 2024-04-24T22:57:23.558 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-04-24T22:57:23.559 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:57:23.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:57:23.559 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-04-24T22:57:23.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-04-24T22:57:23.559 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-04-24T22:57:23.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-04-24T22:57:23.562 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2024-04-24T22:57:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1315630080 bytes 2024-04-24T22:57:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1374265344 bytes 2024-04-24T22:57:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1406089216 bytes 2024-04-24T22:57:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1446188032 bytes 2024-04-24T22:57:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1477702656 bytes 2024-04-24T22:57:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1484602368 bytes 2024-04-24T22:57:24.341 INFO:tasks.ceph.cluster2.osd.0.smithi134.stderr:2024-04-24T22:57:24.339+0000 7fb1e9f85700 -1 reset not still connected to 0x5586d2b82680 2024-04-24T22:57:24.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1537187840 bytes 2024-04-24T22:57:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1588368384 bytes 2024-04-24T22:57:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1588945920 bytes 2024-04-24T22:57:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1659574272 bytes 2024-04-24T22:57:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1705923584 bytes 2024-04-24T22:57:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1755096064 bytes 2024-04-24T22:57:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1836487680 bytes 2024-04-24T22:57:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1929282560 bytes 2024-04-24T22:57:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1966552064 bytes 2024-04-24T22:57:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 2076254208 bytes 2024-04-24T22:57:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 2121257984 bytes 2024-04-24T22:57:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 2157479936 bytes 2024-04-24T22:57:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 2241364992 bytes 2024-04-24T22:57:26.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 2266431488 bytes 2024-04-24T22:57:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 2335778816 bytes 2024-04-24T22:57:26.768 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2348590080 bytes 2024-04-24T22:57:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2405388288 bytes 2024-04-24T22:57:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2424083456 bytes 2024-04-24T22:57:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2481726464 bytes 2024-04-24T22:57:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2548258816 bytes 2024-04-24T22:57:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2571120640 bytes 2024-04-24T22:57:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2684643328 bytes 2024-04-24T22:57:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2694808576 bytes 2024-04-24T22:57:27.963 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2734080000 bytes 2024-04-24T22:57:28.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2772488192 bytes 2024-04-24T22:57:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2809904128 bytes 2024-04-24T22:57:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2815192064 bytes 2024-04-24T22:57:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:57:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:57:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:57:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:57:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2024-04-24T22:57:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:57:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:57:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:57:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=present 2024-04-24T22:57:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:57:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:57:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:57:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_2 2024-04-24T22:57:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:57:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:57:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_2 2024-04-24T22:57:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-24T22:57:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:57:28.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id=10dcaf062a62 2024-04-24T22:57:28.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:57:28.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:57:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_2 present 10dcaf062a62 2024-04-24T22:57:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:57:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:57:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:57:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=present 2024-04-24T22:57:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id=10dcaf062a62 2024-04-24T22:57:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:57:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:57:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_2 2024-04-24T22:57:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:57:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:57:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_2 2024-04-24T22:57:28.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-24T22:57:28.610 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:57:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id=10dcaf062a62 2024-04-24T22:57:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 10dcaf062a62 2024-04-24T22:57:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -z 10dcaf062a62 -o 10dcaf062a62 = 10dcaf062a62 2024-04-24T22:57:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=present 2024-04-24T22:57:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test present = present 2024-04-24T22:57:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:57:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ disable_mirror cluster2 mirror image_2 2024-04-24T22:57:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:57:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:57:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:57:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2024-04-24T22:57:46.691 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3335015712914826) > weight (0.25) 2024-04-24T22:57:46.692 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-04-24T22:57:46.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-04-24T22:57:46.692 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-04-24T22:57:46.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T22:57:46.692+0000 7f7de1562700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 56776) UID: 1000 2024-04-24T22:57:46.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:57:50.693 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-04-24T22:57:50.694 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:57:50.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:57:50.694 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-04-24T22:57:50.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-24T22:57:50.695 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-24T22:57:50.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-24T22:57:50.698 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2024-04-24T22:57:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring disabled 2024-04-24T22:57:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-04-24T22:57:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:57:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:57:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:57:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:57:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:57:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:57:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:57:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_2 2024-04-24T22:57:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:57:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:57:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_2 2024-04-24T22:57:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-24T22:57:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:57:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-04-24T22:57:56.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:57:56.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:57:56.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:57:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-04-24T22:57:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:57:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:57:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:57:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:57:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:57:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:57:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:57:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_2 2024-04-24T22:57:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:57:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:57:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_2 2024-04-24T22:57:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-04-24T22:57:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:57:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-04-24T22:57:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:57:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:57:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:57:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:57:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ purge_snapshots cluster2 mirror image_2 2024-04-24T22:57:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:57:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:57:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:57:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2024-04-24T22:57:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_2 2024-04-24T22:57:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:57:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:57:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:57:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:57:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:57:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_2 2024-04-24T22:57:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:57:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:57:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_2 2024-04-24T22:57:56.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-04-24T22:57:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-04-24T22:57:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-24T22:57:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:57:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:57:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_3 2024-04-24T22:57:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_3 512M 2024-04-24T22:57:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:57:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:57:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:57:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:57:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:57:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:57:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:57:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 512M ']' 2024-04-24T22:57:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=512M 2024-04-24T22:57:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:57:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2024-04-24T22:57:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_3 2024-04-24T22:57:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:57:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:57:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:57:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:57:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-04-24T22:57:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:57:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-04-24T22:57:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_3': 2024-04-24T22:57:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T22:57:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:57:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:57:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 19994a411fce 2024-04-24T22:57:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.19994a411fce 2024-04-24T22:57:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:57:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:57:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:57:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:57:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:57:56 2024 2024-04-24T22:57:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:57:56 2024 2024-04-24T22:57:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:57:56 2024 2024-04-24T22:57:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:57:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:57:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 191fc7e1-12aa-4d75-8079-0f23c3c846a9 2024-04-24T22:57:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:57:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_3 2024-04-24T22:57:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:57:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:57:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:57:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:57:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=14 2024-04-24T22:57:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:57:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:57:57.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 27983872 bytes 2024-04-24T22:57:57.781 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 52767744 bytes 2024-04-24T22:57:57.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 76254208 bytes 2024-04-24T22:57:58.057 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 106277888 bytes 2024-04-24T22:57:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 137202688 bytes 2024-04-24T22:57:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 204203008 bytes 2024-04-24T22:57:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 245079040 bytes 2024-04-24T22:57:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 253850624 bytes 2024-04-24T22:57:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 257767424 bytes 2024-04-24T22:57:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 329739264 bytes 2024-04-24T22:57:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 360731648 bytes 2024-04-24T22:57:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 419703808 bytes 2024-04-24T22:57:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 426710016 bytes 2024-04-24T22:57:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 474474496 bytes 2024-04-24T22:57:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 500718592 bytes 2024-04-24T22:57:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 556343296 bytes 2024-04-24T22:57:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 610481152 bytes 2024-04-24T22:57:59.992 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 627476480 bytes 2024-04-24T22:58:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 664928256 bytes 2024-04-24T22:58:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 724758528 bytes 2024-04-24T22:58:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 744546304 bytes 2024-04-24T22:58:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 750347264 bytes 2024-04-24T22:58:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 832077824 bytes 2024-04-24T22:58:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 848330752 bytes 2024-04-24T22:58:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 918889472 bytes 2024-04-24T22:58:01.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 976843776 bytes 2024-04-24T22:58:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1031853056 bytes 2024-04-24T22:58:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1044654080 bytes 2024-04-24T22:58:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1094838272 bytes 2024-04-24T22:58:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1123361792 bytes 2024-04-24T22:58:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1203746816 bytes 2024-04-24T22:58:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1228326912 bytes 2024-04-24T22:58:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1283453952 bytes 2024-04-24T22:58:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1338089472 bytes 2024-04-24T22:58:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1352280064 bytes 2024-04-24T22:58:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1363096576 bytes 2024-04-24T22:58:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1388193792 bytes 2024-04-24T22:58:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1431021568 bytes 2024-04-24T22:58:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1475272704 bytes 2024-04-24T22:58:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1530869760 bytes 2024-04-24T22:58:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1589291008 bytes 2024-04-24T22:58:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1602764800 bytes 2024-04-24T22:58:03.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1643379712 bytes 2024-04-24T22:58:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1668268032 bytes 2024-04-24T22:58:03.944 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1698757632 bytes 2024-04-24T22:58:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1727362048 bytes 2024-04-24T22:58:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1742269440 bytes 2024-04-24T22:58:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1816974336 bytes 2024-04-24T22:58:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 1848951808 bytes 2024-04-24T22:58:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 1866269696 bytes 2024-04-24T22:58:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 1933901824 bytes 2024-04-24T22:58:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 1942021120 bytes 2024-04-24T22:58:05.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 1965074432 bytes 2024-04-24T22:58:05.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 1965696000 bytes 2024-04-24T22:58:05.278 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2008363008 bytes 2024-04-24T22:58:05.412 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2039346176 bytes 2024-04-24T22:58:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2095058944 bytes 2024-04-24T22:58:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2137448448 bytes 2024-04-24T22:58:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2171346944 bytes 2024-04-24T22:58:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2190028800 bytes 2024-04-24T22:58:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2228065280 bytes 2024-04-24T22:58:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2306043904 bytes 2024-04-24T22:58:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2338598912 bytes 2024-04-24T22:58:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2375461888 bytes 2024-04-24T22:58:06.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2377214976 bytes 2024-04-24T22:58:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2447772672 bytes 2024-04-24T22:58:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2475041792 bytes 2024-04-24T22:58:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2525498368 bytes 2024-04-24T22:58:07.265 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2580329472 bytes 2024-04-24T22:58:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 2612507648 bytes 2024-04-24T22:58:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 2619638784 bytes 2024-04-24T22:58:07.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 2671994880 bytes 2024-04-24T22:58:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 2739671040 bytes 2024-04-24T22:58:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 2808765440 bytes 2024-04-24T22:58:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 2836237312 bytes 2024-04-24T22:58:08.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 2836865024 bytes 2024-04-24T22:58:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 2888263680 bytes 2024-04-24T22:58:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 2913826816 bytes 2024-04-24T22:58:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7900 IOs, 2939202560 bytes 2024-04-24T22:58:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8000 IOs, 3027219456 bytes 2024-04-24T22:58:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8100 IOs, 3064371200 bytes 2024-04-24T22:58:09.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8200 IOs, 3133475840 bytes 2024-04-24T22:58:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8300 IOs, 3145125888 bytes 2024-04-24T22:58:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8400 IOs, 3185597440 bytes 2024-04-24T22:58:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8500 IOs, 3212429312 bytes 2024-04-24T22:58:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8600 IOs, 3250811904 bytes 2024-04-24T22:58:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8700 IOs, 3273598976 bytes 2024-04-24T22:58:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8800 IOs, 3281656832 bytes 2024-04-24T22:58:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8900 IOs, 3305769984 bytes 2024-04-24T22:58:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9000 IOs, 3351997440 bytes 2024-04-24T22:58:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9100 IOs, 3376624640 bytes 2024-04-24T22:58:10.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9200 IOs, 3418181632 bytes 2024-04-24T22:58:10.557 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9300 IOs, 3453113344 bytes 2024-04-24T22:58:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9400 IOs, 3491592192 bytes 2024-04-24T22:58:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9500 IOs, 3544396800 bytes 2024-04-24T22:58:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9600 IOs, 3577694208 bytes 2024-04-24T22:58:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9700 IOs, 3602331648 bytes 2024-04-24T22:58:11.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:9800 IOs, 3613377536 bytes 2024-04-24T22:58:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:58:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:58:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:58:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:58:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2024-04-24T22:58:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:58:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:58:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:58:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=present 2024-04-24T22:58:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:58:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:58:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:58:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_3 2024-04-24T22:58:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:58:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:58:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T22:58:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-24T22:58:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:58:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id=10dcf54b1fa2 2024-04-24T22:58:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:58:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:58:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_3 present 10dcf54b1fa2 2024-04-24T22:58:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:58:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:58:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:58:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=present 2024-04-24T22:58:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id=10dcf54b1fa2 2024-04-24T22:58:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:58:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:58:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_3 2024-04-24T22:58:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:58:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:58:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T22:58:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-24T22:58:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:58:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id=10dcf54b1fa2 2024-04-24T22:58:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 10dcf54b1fa2 2024-04-24T22:58:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -z 10dcf54b1fa2 -o 10dcf54b1fa2 = 10dcf54b1fa2 2024-04-24T22:58:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=present 2024-04-24T22:58:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test present = present 2024-04-24T22:58:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:58:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ disable_mirror cluster2 mirror image_3 2024-04-24T22:58:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:58:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:58:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:58:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2024-04-24T22:58:27.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:58:27.666+0000 7f9902323700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e8149f4680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:58:27.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:58:27.666+0000 7f990832f700 -1 rbd::mirror::ImageReplayer: 0x55e814b80a00 [3/191fc7e1-12aa-4d75-8079-0f23c3c846a9] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:58:42.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T22:58:42.430+0000 7f4a5070b700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55dd06714400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=372 2024-04-24T22:58:42.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T22:58:42.430+0000 7f4a5070b700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55dd06714400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=372, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=d9b73256-b8ac-4751-9b60-95a0234fa37c, primary_snap_id=174, last_copied_object_number=128, snap_seqs={372=18446744073709551614}] 2024-04-24T22:58:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring disabled 2024-04-24T22:58:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-04-24T22:58:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:58:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:58:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:58:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:58:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:58:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:58:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:58:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_3 2024-04-24T22:58:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:58:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:58:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T22:58:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-24T22:58:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:58:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-04-24T22:58:42.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:58:42.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:58:42.489 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:58:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-04-24T22:58:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:58:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:58:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:58:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:58:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:58:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:58:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:58:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_3 2024-04-24T22:58:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:58:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:58:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_3 2024-04-24T22:58:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-04-24T22:58:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:58:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-04-24T22:58:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:58:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:58:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:58:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:58:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ purge_snapshots cluster2 mirror image_3 2024-04-24T22:58:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:58:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:58:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:58:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2024-04-24T22:58:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_3 2024-04-24T22:58:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:58:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:58:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:58:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:58:42.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:58:42.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_3 2024-04-24T22:58:42.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:58:42.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:58:42.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_3 2024-04-24T22:58:42.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-04-24T22:58:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-04-24T22:58:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-24T22:58:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:58:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:58:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_4 2024-04-24T22:58:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_4 512M 2024-04-24T22:58:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:58:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:58:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:58:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:58:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:58:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:58:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:58:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 512M ']' 2024-04-24T22:58:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=512M 2024-04-24T22:58:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:58:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2024-04-24T22:58:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_4 2024-04-24T22:58:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:58:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:58:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:58:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:58:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-04-24T22:58:43.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:58:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-04-24T22:58:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_4': 2024-04-24T22:58:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T22:58:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:58:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:58:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 19b1a10985a7 2024-04-24T22:58:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.19b1a10985a7 2024-04-24T22:58:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:58:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:58:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:58:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:58:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:58:43 2024 2024-04-24T22:58:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:58:43 2024 2024-04-24T22:58:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:58:43 2024 2024-04-24T22:58:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:58:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:58:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 3d923e27-2f31-44a9-b2d2-8d3d94a3d1e4 2024-04-24T22:58:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:58:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_4 2024-04-24T22:58:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:58:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:58:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:58:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:58:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=8 2024-04-24T22:58:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:58:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:58:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 17305600 bytes 2024-04-24T22:58:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 49841152 bytes 2024-04-24T22:58:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 145199104 bytes 2024-04-24T22:58:44.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 214315008 bytes 2024-04-24T22:58:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 279126016 bytes 2024-04-24T22:58:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 339631104 bytes 2024-04-24T22:58:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 383962112 bytes 2024-04-24T22:58:44.865 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 420126720 bytes 2024-04-24T22:58:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 459089920 bytes 2024-04-24T22:58:45.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 481359872 bytes 2024-04-24T22:58:45.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 570939392 bytes 2024-04-24T22:58:45.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 591281152 bytes 2024-04-24T22:58:45.563 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-04-24T22:58:45.564 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-04-24T22:58:45.564 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-04-24T22:58:45.565 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T22:58:45.564+0000 7f9909331700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 64286) UID: 1000 2024-04-24T22:58:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 660549632 bytes 2024-04-24T22:58:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 706469888 bytes 2024-04-24T22:58:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 731746304 bytes 2024-04-24T22:58:45.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:58:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 831034368 bytes 2024-04-24T22:58:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 845408256 bytes 2024-04-24T22:58:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 940307456 bytes 2024-04-24T22:58:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 978007040 bytes 2024-04-24T22:58:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 1034828800 bytes 2024-04-24T22:58:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 1079732224 bytes 2024-04-24T22:58:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 1122530304 bytes 2024-04-24T22:58:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1153817600 bytes 2024-04-24T22:58:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1270045696 bytes 2024-04-24T22:58:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1292021760 bytes 2024-04-24T22:58:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1319126016 bytes 2024-04-24T22:58:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1325999104 bytes 2024-04-24T22:58:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1338613760 bytes 2024-04-24T22:58:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1394662400 bytes 2024-04-24T22:58:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1407948800 bytes 2024-04-24T22:58:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1413437440 bytes 2024-04-24T22:58:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1435241472 bytes 2024-04-24T22:58:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1537802240 bytes 2024-04-24T22:58:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1588496384 bytes 2024-04-24T22:58:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1646345216 bytes 2024-04-24T22:58:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1687653376 bytes 2024-04-24T22:58:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1793419264 bytes 2024-04-24T22:58:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1836032000 bytes 2024-04-24T22:58:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1910061056 bytes 2024-04-24T22:58:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1934159872 bytes 2024-04-24T22:58:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1948474368 bytes 2024-04-24T22:58:49.797 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1949104128 bytes 2024-04-24T22:58:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 2027147264 bytes 2024-04-24T22:58:50.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 2067075072 bytes 2024-04-24T22:58:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 2083530752 bytes 2024-04-24T22:58:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 2110270464 bytes 2024-04-24T22:58:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 2181497856 bytes 2024-04-24T22:58:50.565 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-04-24T22:58:50.565 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:58:50.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:58:50.566 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-04-24T22:58:50.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-24T22:58:50.566 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-24T22:58:50.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-24T22:58:50.569 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-04-24T22:58:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2239748096 bytes 2024-04-24T22:58:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2286472192 bytes 2024-04-24T22:58:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2328145920 bytes 2024-04-24T22:58:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2395497472 bytes 2024-04-24T22:58:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2418350080 bytes 2024-04-24T22:58:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2467047424 bytes 2024-04-24T22:58:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:58:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:58:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:58:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:58:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2024-04-24T22:58:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:58:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:58:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:58:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=present 2024-04-24T22:58:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:58:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:58:51.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:58:51.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_4 2024-04-24T22:58:51.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:58:51.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:58:51.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T22:58:51.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-24T22:58:51.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:58:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id=10dc8efb9faf 2024-04-24T22:58:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:58:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:58:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_4 present 10dc8efb9faf 2024-04-24T22:58:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:58:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:58:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:58:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=present 2024-04-24T22:58:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id=10dc8efb9faf 2024-04-24T22:58:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:58:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:58:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_4 2024-04-24T22:58:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:58:51.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:58:51.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T22:58:51.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-24T22:58:51.626 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:58:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id=10dc8efb9faf 2024-04-24T22:58:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 10dc8efb9faf 2024-04-24T22:58:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -z 10dc8efb9faf -o 10dc8efb9faf = 10dc8efb9faf 2024-04-24T22:58:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=present 2024-04-24T22:58:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test present = present 2024-04-24T22:58:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:58:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ disable_mirror cluster2 mirror image_4 2024-04-24T22:58:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:58:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:58:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:58:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2024-04-24T22:58:53.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:58:53.832+0000 7f7746536700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c81fdd9520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:58:53.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:58:53.832+0000 7f774c542700 -1 rbd::mirror::ImageReplayer: 0x55c81cc4d680 [3/3d923e27-2f31-44a9-b2d2-8d3d94a3d1e4] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T22:59:05.699 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7166796496831351) > weight (0.25) 2024-04-24T22:59:05.699 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-04-24T22:59:05.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-04-24T22:59:05.700 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-04-24T22:59:05.700 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-04-24T22:59:05.700 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T22:59:05.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T22:59:05.699+0000 7f9933717700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 81644) UID: 1000 2024-04-24T22:59:05.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T22:59:11.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T22:59:11.802 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-04-24T22:59:11.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-24T22:59:11.802 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-24T22:59:11.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-24T22:59:11.806 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-04-24T22:59:20.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T22:59:20.922+0000 7f4a51f0e700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55dd03e19800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=374 2024-04-24T22:59:20.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T22:59:20.922+0000 7f4a51f0e700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55dd03e19800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=374, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=d9b73256-b8ac-4751-9b60-95a0234fa37c, primary_snap_id=176, last_copied_object_number=128, snap_seqs={374=18446744073709551614}] 2024-04-24T22:59:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring disabled 2024-04-24T22:59:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-04-24T22:59:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:59:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:59:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:59:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T22:59:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:59:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:59:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:59:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_4 2024-04-24T22:59:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:59:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:59:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T22:59:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-24T22:59:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:59:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-04-24T22:59:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T22:59:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:59:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:59:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-04-24T22:59:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:59:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:59:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:59:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T22:59:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:59:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:59:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:59:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_4 2024-04-24T22:59:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:59:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:59:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_4 2024-04-24T22:59:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-04-24T22:59:21.090 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:59:21.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-04-24T22:59:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T22:59:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:59:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T22:59:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:59:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ purge_snapshots cluster2 mirror image_4 2024-04-24T22:59:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:59:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:59:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:59:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2024-04-24T22:59:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_4 2024-04-24T22:59:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:59:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:59:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:59:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T22:59:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T22:59:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_4 2024-04-24T22:59:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:59:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:59:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_4 2024-04-24T22:59:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-04-24T22:59:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-04-24T22:59:21.475 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-24T22:59:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:59:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T22:59:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_5 2024-04-24T22:59:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_5 512M 2024-04-24T22:59:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:59:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:59:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:59:21.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:59:21.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:59:21.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:59:21.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T22:59:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 512M ']' 2024-04-24T22:59:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=512M 2024-04-24T22:59:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T22:59:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2024-04-24T22:59:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_5 2024-04-24T22:59:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:59:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:59:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:59:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T22:59:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-04-24T22:59:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T22:59:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-04-24T22:59:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_5': 2024-04-24T22:59:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T22:59:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T22:59:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T22:59:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 19d6c1bc1732 2024-04-24T22:59:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.19d6c1bc1732 2024-04-24T22:59:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T22:59:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T22:59:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T22:59:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T22:59:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 22:59:21 2024 2024-04-24T22:59:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 22:59:21 2024 2024-04-24T22:59:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 22:59:21 2024 2024-04-24T22:59:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T22:59:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T22:59:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: db7ee67d-1011-4dfe-bec5-eb16caa0bcee 2024-04-24T22:59:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T22:59:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_5 2024-04-24T22:59:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:59:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:59:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:59:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T22:59:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=10 2024-04-24T22:59:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T22:59:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-04-24T22:59:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 41495552 bytes 2024-04-24T22:59:22.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 42103808 bytes 2024-04-24T22:59:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 58058752 bytes 2024-04-24T22:59:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 108995584 bytes 2024-04-24T22:59:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 136211456 bytes 2024-04-24T22:59:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 186880000 bytes 2024-04-24T22:59:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 242904064 bytes 2024-04-24T22:59:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 309868544 bytes 2024-04-24T22:59:23.603 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 381664256 bytes 2024-04-24T22:59:23.722 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 393744384 bytes 2024-04-24T22:59:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 456672256 bytes 2024-04-24T22:59:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 477298688 bytes 2024-04-24T22:59:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 533068800 bytes 2024-04-24T22:59:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 556081152 bytes 2024-04-24T22:59:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 580379648 bytes 2024-04-24T22:59:24.544 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 634994688 bytes 2024-04-24T22:59:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 719015936 bytes 2024-04-24T22:59:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 765963264 bytes 2024-04-24T22:59:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 773956608 bytes 2024-04-24T22:59:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 825302016 bytes 2024-04-24T22:59:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 915017728 bytes 2024-04-24T22:59:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 971485184 bytes 2024-04-24T22:59:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1023961088 bytes 2024-04-24T22:59:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1030249472 bytes 2024-04-24T22:59:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1072989184 bytes 2024-04-24T22:59:25.964 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1077474304 bytes 2024-04-24T22:59:26.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1119694848 bytes 2024-04-24T22:59:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1209979904 bytes 2024-04-24T22:59:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1210568704 bytes 2024-04-24T22:59:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1245688832 bytes 2024-04-24T22:59:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1283311616 bytes 2024-04-24T22:59:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1309582336 bytes 2024-04-24T22:59:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1356326912 bytes 2024-04-24T22:59:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1397512192 bytes 2024-04-24T22:59:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1419097088 bytes 2024-04-24T22:59:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1428987904 bytes 2024-04-24T22:59:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1479559168 bytes 2024-04-24T22:59:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1518407680 bytes 2024-04-24T22:59:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1530771456 bytes 2024-04-24T22:59:27.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1567293440 bytes 2024-04-24T22:59:27.917 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1572916224 bytes 2024-04-24T22:59:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1615390720 bytes 2024-04-24T22:59:28.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1722426368 bytes 2024-04-24T22:59:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1780028416 bytes 2024-04-24T22:59:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1831109632 bytes 2024-04-24T22:59:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1934716928 bytes 2024-04-24T22:59:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1955578880 bytes 2024-04-24T22:59:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1963609088 bytes 2024-04-24T22:59:29.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2005241856 bytes 2024-04-24T22:59:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2041246720 bytes 2024-04-24T22:59:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2066516992 bytes 2024-04-24T22:59:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2096131072 bytes 2024-04-24T22:59:29.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2128560128 bytes 2024-04-24T22:59:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2164323328 bytes 2024-04-24T22:59:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2218946560 bytes 2024-04-24T22:59:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2220297216 bytes 2024-04-24T22:59:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2278297600 bytes 2024-04-24T22:59:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2316375040 bytes 2024-04-24T22:59:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2347736064 bytes 2024-04-24T22:59:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2397298688 bytes 2024-04-24T22:59:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2481675264 bytes 2024-04-24T22:59:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2566532096 bytes 2024-04-24T22:59:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2631799808 bytes 2024-04-24T22:59:31.282 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2675875840 bytes 2024-04-24T22:59:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2704714752 bytes 2024-04-24T22:59:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2729221120 bytes 2024-04-24T22:59:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2817631232 bytes 2024-04-24T22:59:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2868562944 bytes 2024-04-24T22:59:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2869186560 bytes 2024-04-24T22:59:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T22:59:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T22:59:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T22:59:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:59:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2024-04-24T22:59:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:59:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:59:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:59:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=present 2024-04-24T22:59:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T22:59:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T22:59:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T22:59:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_5 2024-04-24T22:59:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:59:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:59:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_5 2024-04-24T22:59:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-24T22:59:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:59:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id=10dcebfee896 2024-04-24T22:59:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T22:59:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T22:59:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_5 present 10dcebfee896 2024-04-24T22:59:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T22:59:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:59:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:59:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=present 2024-04-24T22:59:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id=10dcebfee896 2024-04-24T22:59:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T22:59:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T22:59:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_5 2024-04-24T22:59:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T22:59:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T22:59:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_5 2024-04-24T22:59:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-24T22:59:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T22:59:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id=10dcebfee896 2024-04-24T22:59:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 10dcebfee896 2024-04-24T22:59:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -z 10dcebfee896 -o 10dcebfee896 = 10dcebfee896 2024-04-24T22:59:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=present 2024-04-24T22:59:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test present = present 2024-04-24T22:59:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T22:59:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ disable_mirror cluster2 mirror image_5 2024-04-24T22:59:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T22:59:32.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T22:59:32.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T22:59:32.160 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2024-04-24T22:59:53.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:59:53.832+0000 7f7745d35700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c81fdd5520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T22:59:53.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T22:59:53.832+0000 7f774c542700 -1 rbd::mirror::ImageReplayer: 0x55c81cc4d400 [3/db7ee67d-1011-4dfe-bec5-eb16caa0bcee] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T23:00:02.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:00:02.377+0000 7f4a5070b700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55dd031e8400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=376 2024-04-24T23:00:02.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:00:02.377+0000 7f4a5070b700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55dd031e8400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=376, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=d9b73256-b8ac-4751-9b60-95a0234fa37c, primary_snap_id=178, last_copied_object_number=128, snap_seqs={376=18446744073709551614}] 2024-04-24T23:00:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring disabled 2024-04-24T23:00:02.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:00:02.400+0000 7f4a5070b700 -1 rbd::mirror::ImageReplayer: 0x55dd03962780 [3/db7ee67d-1011-4dfe-bec5-eb16caa0bcee] start: already running 2024-04-24T23:00:02.409 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-04-24T23:00:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:00:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:00:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T23:00:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T23:00:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T23:00:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T23:00:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T23:00:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_5 2024-04-24T23:00:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:00:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:00:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_5 2024-04-24T23:00:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-24T23:00:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T23:00:02.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-04-24T23:00:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T23:00:02.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T23:00:02.445 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T23:00:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-04-24T23:00:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:00:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:00:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T23:00:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T23:00:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T23:00:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T23:00:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T23:00:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_5 2024-04-24T23:00:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:00:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:00:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_5 2024-04-24T23:00:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-04-24T23:00:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T23:00:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-04-24T23:00:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T23:00:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T23:00:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T23:00:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:00:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ purge_snapshots cluster2 mirror image_5 2024-04-24T23:00:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:00:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:00:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T23:00:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2024-04-24T23:00:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_5 2024-04-24T23:00:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:00:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:00:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T23:00:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T23:00:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T23:00:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_5 2024-04-24T23:00:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:00:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:00:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_5 2024-04-24T23:00:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-04-24T23:00:02.658 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-04-24T23:00:02.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-24T23:00:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:00:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T23:00:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_6 2024-04-24T23:00:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_6 512M 2024-04-24T23:00:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:00:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T23:00:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:00:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T23:00:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T23:00:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T23:00:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T23:00:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 512M ']' 2024-04-24T23:00:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=512M 2024-04-24T23:00:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T23:00:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2024-04-24T23:00:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_6 2024-04-24T23:00:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:00:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:00:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T23:00:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T23:00:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-04-24T23:00:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T23:00:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-04-24T23:00:03.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_6': 2024-04-24T23:00:03.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T23:00:03.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T23:00:03.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T23:00:03.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 19ee49482438 2024-04-24T23:00:03.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.19ee49482438 2024-04-24T23:00:03.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T23:00:03.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T23:00:03.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T23:00:03.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T23:00:03.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 23:00:02 2024 2024-04-24T23:00:03.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 23:00:02 2024 2024-04-24T23:00:03.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 23:00:02 2024 2024-04-24T23:00:03.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T23:00:03.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T23:00:03.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 6778e6e7-66f9-4c96-8835-6e48c10d5895 2024-04-24T23:00:03.430 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T23:00:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_6 2024-04-24T23:00:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:00:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:00:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T23:00:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T23:00:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=10 2024-04-24T23:00:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T23:00:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-04-24T23:00:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 604160 bytes 2024-04-24T23:00:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 1219584 bytes 2024-04-24T23:00:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 18506752 bytes 2024-04-24T23:00:04.125 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 108932096 bytes 2024-04-24T23:00:04.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 156261376 bytes 2024-04-24T23:00:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 235724800 bytes 2024-04-24T23:00:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 295264256 bytes 2024-04-24T23:00:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 296502272 bytes 2024-04-24T23:00:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 320552960 bytes 2024-04-24T23:00:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 321182720 bytes 2024-04-24T23:00:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 362274816 bytes 2024-04-24T23:00:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 400017408 bytes 2024-04-24T23:00:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 400829440 bytes 2024-04-24T23:00:05.441 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 438509568 bytes 2024-04-24T23:00:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 482617344 bytes 2024-04-24T23:00:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 503921664 bytes 2024-04-24T23:00:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 517066752 bytes 2024-04-24T23:00:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 571264000 bytes 2024-04-24T23:00:06.097 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 599258112 bytes 2024-04-24T23:00:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 706375680 bytes 2024-04-24T23:00:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 729529344 bytes 2024-04-24T23:00:06.556 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 764956672 bytes 2024-04-24T23:00:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 829774848 bytes 2024-04-24T23:00:06.851 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 869043200 bytes 2024-04-24T23:00:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 870526976 bytes 2024-04-24T23:00:07.087 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 905732096 bytes 2024-04-24T23:00:07.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 953749504 bytes 2024-04-24T23:00:07.393 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1017377792 bytes 2024-04-24T23:00:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1051813888 bytes 2024-04-24T23:00:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1068895232 bytes 2024-04-24T23:00:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1090444288 bytes 2024-04-24T23:00:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1151631360 bytes 2024-04-24T23:00:08.051 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1180062720 bytes 2024-04-24T23:00:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1221614592 bytes 2024-04-24T23:00:08.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1222237184 bytes 2024-04-24T23:00:08.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1222859776 bytes 2024-04-24T23:00:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1237268480 bytes 2024-04-24T23:00:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1263406080 bytes 2024-04-24T23:00:08.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1326158848 bytes 2024-04-24T23:00:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1391389696 bytes 2024-04-24T23:00:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1392858112 bytes 2024-04-24T23:00:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1403163648 bytes 2024-04-24T23:00:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1483499520 bytes 2024-04-24T23:00:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1505031168 bytes 2024-04-24T23:00:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1546449920 bytes 2024-04-24T23:00:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1617252352 bytes 2024-04-24T23:00:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1669398528 bytes 2024-04-24T23:00:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1755499520 bytes 2024-04-24T23:00:10.424 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 1772438528 bytes 2024-04-24T23:00:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 1839098880 bytes 2024-04-24T23:00:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 1842246656 bytes 2024-04-24T23:00:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 1912535040 bytes 2024-04-24T23:00:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 1933432832 bytes 2024-04-24T23:00:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2045194240 bytes 2024-04-24T23:00:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2095284224 bytes 2024-04-24T23:00:11.347 INFO:tasks.ceph.cluster2.osd.1.smithi134.stderr:2024-04-24T23:00:11.345+0000 7f05853c0700 -1 reset not still connected to 0x5619cc1ec8f0 2024-04-24T23:00:11.454 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2114422784 bytes 2024-04-24T23:00:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2140700672 bytes 2024-04-24T23:00:11.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2194843648 bytes 2024-04-24T23:00:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2198074368 bytes 2024-04-24T23:00:12.007 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2276065280 bytes 2024-04-24T23:00:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2290306048 bytes 2024-04-24T23:00:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2371216384 bytes 2024-04-24T23:00:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2437367808 bytes 2024-04-24T23:00:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2449693696 bytes 2024-04-24T23:00:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2513011712 bytes 2024-04-24T23:00:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2552948736 bytes 2024-04-24T23:00:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2569970688 bytes 2024-04-24T23:00:13.117 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2610063360 bytes 2024-04-24T23:00:13.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2633871360 bytes 2024-04-24T23:00:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 2665543680 bytes 2024-04-24T23:00:13.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T23:00:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T23:00:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T23:00:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:00:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2024-04-24T23:00:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:00:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:00:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T23:00:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=present 2024-04-24T23:00:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T23:00:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T23:00:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T23:00:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_6 2024-04-24T23:00:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:00:13.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:00:13.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T23:00:13.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-24T23:00:13.444 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T23:00:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id=10dcd024fed0 2024-04-24T23:00:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T23:00:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T23:00:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_6 present 10dcd024fed0 2024-04-24T23:00:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:00:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:00:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T23:00:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=present 2024-04-24T23:00:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id=10dcd024fed0 2024-04-24T23:00:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T23:00:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T23:00:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_6 2024-04-24T23:00:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:00:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:00:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T23:00:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-24T23:00:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T23:00:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id=10dcd024fed0 2024-04-24T23:00:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 10dcd024fed0 2024-04-24T23:00:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -z 10dcd024fed0 -o 10dcd024fed0 = 10dcd024fed0 2024-04-24T23:00:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=present 2024-04-24T23:00:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test present = present 2024-04-24T23:00:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:00:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ disable_mirror cluster2 mirror image_6 2024-04-24T23:00:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:00:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:00:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T23:00:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2024-04-24T23:00:20.806 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.737914331813073) > weight (0.25) 2024-04-24T23:00:20.807 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6501431519169556) > weight (0.25) 2024-04-24T23:00:20.807 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7722823559915383) > weight (0.25) 2024-04-24T23:00:20.807 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-04-24T23:00:20.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-04-24T23:00:20.807 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-04-24T23:00:20.807 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-04-24T23:00:20.807 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T23:00:20.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:00:20.807+0000 7f4a5df26700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 47994) UID: 1000 2024-04-24T23:00:21.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T23:00:23.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:00:23.833+0000 7f7745d35700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c81e3f1380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T23:00:23.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:00:23.833+0000 7f774c542700 -1 rbd::mirror::ImageReplayer: 0x55c81cc4db80 [3/6778e6e7-66f9-4c96-8835-6e48c10d5895] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T23:00:24.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:00:24.901+0000 7f7746536700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c81feb8680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T23:00:24.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:00:24.901+0000 7f774c542700 -1 rbd::mirror::ImageReplayer: 0x55c81cc4db80 [3/6778e6e7-66f9-4c96-8835-6e48c10d5895] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T23:00:26.570 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5484003607741588) > weight (0.25) 2024-04-24T23:00:26.570 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.40121211046328475) > weight (0.25) 2024-04-24T23:00:26.570 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7923420737864886) > weight (0.25) 2024-04-24T23:00:26.570 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-04-24T23:00:26.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-04-24T23:00:26.571 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-04-24T23:00:26.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:00:26.570+0000 7f8de8442700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 60550) UID: 1000 2024-04-24T23:00:26.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T23:00:26.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:00:26.909 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-04-24T23:00:26.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-04-24T23:00:26.910 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-04-24T23:00:26.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-04-24T23:00:26.912 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2024-04-24T23:00:29.571 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-04-24T23:00:29.572 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T23:00:29.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:00:29.572 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-04-24T23:00:29.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-04-24T23:00:29.572 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-04-24T23:00:29.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-04-24T23:00:29.575 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2024-04-24T23:00:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring disabled 2024-04-24T23:00:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-04-24T23:00:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:00:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:00:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T23:00:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T23:00:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T23:00:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T23:00:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T23:00:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_6 2024-04-24T23:00:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:00:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:00:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T23:00:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-24T23:00:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T23:00:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-04-24T23:00:42.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T23:00:42.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T23:00:42.504 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T23:00:42.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-04-24T23:00:42.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:00:42.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:00:42.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T23:00:42.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T23:00:42.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T23:00:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T23:00:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T23:00:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_6 2024-04-24T23:00:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:00:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:00:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_6 2024-04-24T23:00:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-04-24T23:00:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T23:00:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-04-24T23:00:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T23:00:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T23:00:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T23:00:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:00:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ purge_snapshots cluster2 mirror image_6 2024-04-24T23:00:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:00:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:00:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T23:00:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2024-04-24T23:00:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_6 2024-04-24T23:00:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:00:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:00:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T23:00:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T23:00:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T23:00:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_6 2024-04-24T23:00:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:00:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:00:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_6 2024-04-24T23:00:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-04-24T23:00:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-04-24T23:00:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-24T23:00:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:00:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T23:00:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_7 2024-04-24T23:00:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_7 512M 2024-04-24T23:00:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:00:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T23:00:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:00:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T23:00:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T23:00:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T23:00:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T23:00:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 512M ']' 2024-04-24T23:00:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=512M 2024-04-24T23:00:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T23:00:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2024-04-24T23:00:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_7 2024-04-24T23:00:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:00:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:00:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T23:00:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T23:00:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-04-24T23:00:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T23:00:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-04-24T23:00:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_7': 2024-04-24T23:00:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T23:00:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T23:00:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T23:00:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 1a1489532eb4 2024-04-24T23:00:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.1a1489532eb4 2024-04-24T23:00:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T23:00:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T23:00:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T23:00:43.498 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T23:00:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 23:00:43 2024 2024-04-24T23:00:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 23:00:43 2024 2024-04-24T23:00:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 23:00:43 2024 2024-04-24T23:00:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T23:00:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T23:00:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 10edd616-064d-469e-bbd1-826a02be89fd 2024-04-24T23:00:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T23:00:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_7 2024-04-24T23:00:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:00:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:00:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T23:00:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T23:00:43.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=11 2024-04-24T23:00:43.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T23:00:43.502 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-04-24T23:00:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 14954496 bytes 2024-04-24T23:00:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 62016512 bytes 2024-04-24T23:00:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 147964928 bytes 2024-04-24T23:00:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 186803200 bytes 2024-04-24T23:00:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 261238784 bytes 2024-04-24T23:00:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 282582016 bytes 2024-04-24T23:00:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 288444416 bytes 2024-04-24T23:00:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 307555328 bytes 2024-04-24T23:00:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 392840192 bytes 2024-04-24T23:00:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 420543488 bytes 2024-04-24T23:00:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 445935616 bytes 2024-04-24T23:00:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 453118976 bytes 2024-04-24T23:00:45.403 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 465670144 bytes 2024-04-24T23:00:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 503031808 bytes 2024-04-24T23:00:45.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 522814464 bytes 2024-04-24T23:00:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 573803520 bytes 2024-04-24T23:00:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 601289728 bytes 2024-04-24T23:00:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 647405568 bytes 2024-04-24T23:00:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 687174656 bytes 2024-04-24T23:00:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 719877120 bytes 2024-04-24T23:00:46.516 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 786817024 bytes 2024-04-24T23:00:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 859460608 bytes 2024-04-24T23:00:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 963544064 bytes 2024-04-24T23:00:47.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 986222592 bytes 2024-04-24T23:00:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1032973312 bytes 2024-04-24T23:00:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1041823744 bytes 2024-04-24T23:00:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1067038720 bytes 2024-04-24T23:00:47.613 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1175678976 bytes 2024-04-24T23:00:47.744 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1213274112 bytes 2024-04-24T23:00:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1245879296 bytes 2024-04-24T23:00:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1342002176 bytes 2024-04-24T23:00:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1377537024 bytes 2024-04-24T23:00:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1399313408 bytes 2024-04-24T23:00:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1466604544 bytes 2024-04-24T23:00:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1547309056 bytes 2024-04-24T23:00:48.786 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1565684736 bytes 2024-04-24T23:00:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1619253248 bytes 2024-04-24T23:00:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1686398976 bytes 2024-04-24T23:00:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1691100160 bytes 2024-04-24T23:00:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1706612736 bytes 2024-04-24T23:00:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1729390592 bytes 2024-04-24T23:00:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1791974400 bytes 2024-04-24T23:00:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1793098752 bytes 2024-04-24T23:00:49.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1818480640 bytes 2024-04-24T23:00:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1877835776 bytes 2024-04-24T23:00:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1928230912 bytes 2024-04-24T23:00:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1979462656 bytes 2024-04-24T23:00:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1981153280 bytes 2024-04-24T23:00:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2010768384 bytes 2024-04-24T23:00:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2021896192 bytes 2024-04-24T23:00:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2115284992 bytes 2024-04-24T23:00:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2153992192 bytes 2024-04-24T23:00:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2181295104 bytes 2024-04-24T23:00:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2193311744 bytes 2024-04-24T23:00:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2212315136 bytes 2024-04-24T23:00:51.599 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2216690688 bytes 2024-04-24T23:00:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2322228224 bytes 2024-04-24T23:00:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2322836480 bytes 2024-04-24T23:00:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2353918976 bytes 2024-04-24T23:00:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2444614656 bytes 2024-04-24T23:00:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2475394048 bytes 2024-04-24T23:00:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2483856384 bytes 2024-04-24T23:00:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2553398272 bytes 2024-04-24T23:00:52.752 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2556366848 bytes 2024-04-24T23:00:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2576010240 bytes 2024-04-24T23:00:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2580633600 bytes 2024-04-24T23:00:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2622156800 bytes 2024-04-24T23:00:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2687951872 bytes 2024-04-24T23:00:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2785661952 bytes 2024-04-24T23:00:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 2862573568 bytes 2024-04-24T23:00:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 2904970240 bytes 2024-04-24T23:00:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 2971260928 bytes 2024-04-24T23:00:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 3054030848 bytes 2024-04-24T23:00:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 3063756800 bytes 2024-04-24T23:00:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 3125729280 bytes 2024-04-24T23:00:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T23:00:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T23:00:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T23:00:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:00:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2024-04-24T23:00:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:00:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:00:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T23:00:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=present 2024-04-24T23:00:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T23:00:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T23:00:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T23:00:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_7 2024-04-24T23:00:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:00:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:00:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T23:00:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-24T23:00:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T23:00:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id=1616cbfb80b3 2024-04-24T23:00:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T23:00:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T23:00:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_7 present 1616cbfb80b3 2024-04-24T23:00:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:00:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:00:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T23:00:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=present 2024-04-24T23:00:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id=1616cbfb80b3 2024-04-24T23:00:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T23:00:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T23:00:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_7 2024-04-24T23:00:54.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:00:54.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:00:54.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T23:00:54.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-24T23:00:54.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T23:00:54.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id=1616cbfb80b3 2024-04-24T23:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1616cbfb80b3 2024-04-24T23:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -z 1616cbfb80b3 -o 1616cbfb80b3 = 1616cbfb80b3 2024-04-24T23:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=present 2024-04-24T23:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test present = present 2024-04-24T23:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ disable_mirror cluster2 mirror image_7 2024-04-24T23:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T23:00:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2024-04-24T23:01:23.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:01:23.833+0000 7f7746536700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c81fe87860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T23:01:23.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:01:23.833+0000 7f774cd43700 -1 rbd::mirror::ImageReplayer: 0x55c81cc4db80 [3/10edd616-064d-469e-bbd1-826a02be89fd] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T23:01:24.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:01:24.622+0000 7f572a7fb700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55db40986c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=380 2024-04-24T23:01:24.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:01:24.622+0000 7f572a7fb700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55db40986c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=380, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=d9b73256-b8ac-4751-9b60-95a0234fa37c, primary_snap_id=17c, last_copied_object_number=128, snap_seqs={380=18446744073709551614}] 2024-04-24T23:01:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring disabled 2024-04-24T23:01:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-04-24T23:01:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:01:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:01:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T23:01:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T23:01:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T23:01:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T23:01:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T23:01:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_7 2024-04-24T23:01:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:01:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:01:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T23:01:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-24T23:01:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T23:01:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-04-24T23:01:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T23:01:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T23:01:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T23:01:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-04-24T23:01:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:01:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:01:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T23:01:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T23:01:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T23:01:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T23:01:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T23:01:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_7 2024-04-24T23:01:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:01:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:01:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_7 2024-04-24T23:01:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-04-24T23:01:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T23:01:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-04-24T23:01:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T23:01:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T23:01:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T23:01:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:01:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ purge_snapshots cluster2 mirror image_7 2024-04-24T23:01:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:01:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:01:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T23:01:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2024-04-24T23:01:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_7 2024-04-24T23:01:24.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:01:24.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:01:24.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T23:01:24.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T23:01:24.867 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T23:01:24.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_7 2024-04-24T23:01:24.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:01:24.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:01:24.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_7 2024-04-24T23:01:24.868 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-04-24T23:01:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-04-24T23:01:25.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-24T23:01:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:01:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T23:01:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_8 2024-04-24T23:01:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_8 512M 2024-04-24T23:01:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:01:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T23:01:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:01:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T23:01:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T23:01:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T23:01:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T23:01:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 512M ']' 2024-04-24T23:01:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=512M 2024-04-24T23:01:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T23:01:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2024-04-24T23:01:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_8 2024-04-24T23:01:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:01:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:01:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T23:01:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T23:01:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-04-24T23:01:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T23:01:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-04-24T23:01:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_8': 2024-04-24T23:01:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T23:01:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T23:01:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T23:01:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 1a2c301b2a3e 2024-04-24T23:01:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.1a2c301b2a3e 2024-04-24T23:01:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T23:01:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T23:01:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T23:01:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T23:01:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 23:01:25 2024 2024-04-24T23:01:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 23:01:25 2024 2024-04-24T23:01:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 23:01:25 2024 2024-04-24T23:01:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T23:01:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T23:01:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: 609ae9c2-2dd8-4538-934b-2f9a41b2e0e1 2024-04-24T23:01:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T23:01:25.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_8 2024-04-24T23:01:25.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:01:25.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:01:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T23:01:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T23:01:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=12 2024-04-24T23:01:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T23:01:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-04-24T23:01:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 110164992 bytes 2024-04-24T23:01:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 149727232 bytes 2024-04-24T23:01:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 167369728 bytes 2024-04-24T23:01:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 187833344 bytes 2024-04-24T23:01:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 208774144 bytes 2024-04-24T23:01:26.681 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 243759104 bytes 2024-04-24T23:01:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 283771904 bytes 2024-04-24T23:01:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 325412864 bytes 2024-04-24T23:01:27.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 351289344 bytes 2024-04-24T23:01:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 366541824 bytes 2024-04-24T23:01:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 415772672 bytes 2024-04-24T23:01:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 416368640 bytes 2024-04-24T23:01:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 416958464 bytes 2024-04-24T23:01:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 458013696 bytes 2024-04-24T23:01:27.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 582142976 bytes 2024-04-24T23:01:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 640777216 bytes 2024-04-24T23:01:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 683826176 bytes 2024-04-24T23:01:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 699062272 bytes 2024-04-24T23:01:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 702662656 bytes 2024-04-24T23:01:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 772869120 bytes 2024-04-24T23:01:28.674 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 804892672 bytes 2024-04-24T23:01:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 839077888 bytes 2024-04-24T23:01:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 848825344 bytes 2024-04-24T23:01:29.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 930152448 bytes 2024-04-24T23:01:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 955444224 bytes 2024-04-24T23:01:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 990517248 bytes 2024-04-24T23:01:29.568 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1089432576 bytes 2024-04-24T23:01:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1106060288 bytes 2024-04-24T23:01:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1169094656 bytes 2024-04-24T23:01:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1190648832 bytes 2024-04-24T23:01:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1219146752 bytes 2024-04-24T23:01:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1316674560 bytes 2024-04-24T23:01:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1397488640 bytes 2024-04-24T23:01:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1425830912 bytes 2024-04-24T23:01:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1460686848 bytes 2024-04-24T23:01:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1547899904 bytes 2024-04-24T23:01:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1605610496 bytes 2024-04-24T23:01:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1642730496 bytes 2024-04-24T23:01:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1706869760 bytes 2024-04-24T23:01:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1748501504 bytes 2024-04-24T23:01:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1811511296 bytes 2024-04-24T23:01:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1904606208 bytes 2024-04-24T23:01:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1976282112 bytes 2024-04-24T23:01:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1991859200 bytes 2024-04-24T23:01:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 2072545280 bytes 2024-04-24T23:01:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 2123027456 bytes 2024-04-24T23:01:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 2175183872 bytes 2024-04-24T23:01:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2198177792 bytes 2024-04-24T23:01:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2242878464 bytes 2024-04-24T23:01:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2307358720 bytes 2024-04-24T23:01:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2336215040 bytes 2024-04-24T23:01:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2390417408 bytes 2024-04-24T23:01:33.479 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2414522368 bytes 2024-04-24T23:01:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2445573120 bytes 2024-04-24T23:01:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2502896640 bytes 2024-04-24T23:01:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2527258624 bytes 2024-04-24T23:01:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2604147712 bytes 2024-04-24T23:01:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2642229248 bytes 2024-04-24T23:01:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2691919872 bytes 2024-04-24T23:01:34.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2711689216 bytes 2024-04-24T23:01:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2773606400 bytes 2024-04-24T23:01:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2804506624 bytes 2024-04-24T23:01:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2857931776 bytes 2024-04-24T23:01:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2930312192 bytes 2024-04-24T23:01:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2993505280 bytes 2024-04-24T23:01:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 3015543808 bytes 2024-04-24T23:01:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 3073487872 bytes 2024-04-24T23:01:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 3140222976 bytes 2024-04-24T23:01:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 3157504000 bytes 2024-04-24T23:01:35.969 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 3181043712 bytes 2024-04-24T23:01:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 3211702272 bytes 2024-04-24T23:01:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 3235196928 bytes 2024-04-24T23:01:36.352 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 3262746624 bytes 2024-04-24T23:01:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 3297737728 bytes 2024-04-24T23:01:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 3368218624 bytes 2024-04-24T23:01:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 3432265728 bytes 2024-04-24T23:01:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 3508771840 bytes 2024-04-24T23:01:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 3578545152 bytes 2024-04-24T23:01:37.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7900 IOs, 3613391872 bytes 2024-04-24T23:01:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8000 IOs, 3642518528 bytes 2024-04-24T23:01:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8100 IOs, 3700455424 bytes 2024-04-24T23:01:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T23:01:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T23:01:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T23:01:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:01:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2024-04-24T23:01:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:01:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:01:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T23:01:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=present 2024-04-24T23:01:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T23:01:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T23:01:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T23:01:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_8 2024-04-24T23:01:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:01:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:01:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T23:01:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-24T23:01:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T23:01:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id=1616e390089e 2024-04-24T23:01:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T23:01:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T23:01:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_8 present 1616e390089e 2024-04-24T23:01:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:01:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:01:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T23:01:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=present 2024-04-24T23:01:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id=1616e390089e 2024-04-24T23:01:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T23:01:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T23:01:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_8 2024-04-24T23:01:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:01:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:01:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T23:01:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-24T23:01:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T23:01:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id=1616e390089e 2024-04-24T23:01:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1616e390089e 2024-04-24T23:01:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -z 1616e390089e -o 1616e390089e = 1616e390089e 2024-04-24T23:01:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=present 2024-04-24T23:01:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test present = present 2024-04-24T23:01:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:01:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ disable_mirror cluster2 mirror image_8 2024-04-24T23:01:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:01:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:01:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T23:01:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2024-04-24T23:01:39.913 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9692502862604181) > weight (0.25) 2024-04-24T23:01:39.913 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.46768536870332367) > weight (0.25) 2024-04-24T23:01:39.913 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-04-24T23:01:39.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-04-24T23:01:39.914 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-04-24T23:01:39.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:01:39.914+0000 7f450f3fe700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 74436) UID: 1000 2024-04-24T23:01:40.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T23:01:41.915 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-04-24T23:01:41.915 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T23:01:41.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:01:41.916 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-04-24T23:01:41.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-04-24T23:01:41.916 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-04-24T23:01:41.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-04-24T23:01:41.919 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2024-04-24T23:01:50.576 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7512344879787085) > weight (0.25) 2024-04-24T23:01:50.576 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.34016512501337726) > weight (0.25) 2024-04-24T23:01:50.576 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8445817573246036) > weight (0.25) 2024-04-24T23:01:50.576 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5216678165208012) > weight (0.25) 2024-04-24T23:01:50.576 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2024-04-24T23:01:53.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:01:53.833+0000 7f7746536700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c820092000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T23:01:53.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:01:53.833+0000 7f774c542700 -1 rbd::mirror::ImageReplayer: 0x55c81cc4d680 [3/609ae9c2-2dd8-4538-934b-2f9a41b2e0e1] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T23:02:08.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:02:08.459+0000 7f572a7fb700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55db40daa400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=382 2024-04-24T23:02:08.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:02:08.459+0000 7f572a7fb700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55db40daa400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=382, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=d9b73256-b8ac-4751-9b60-95a0234fa37c, primary_snap_id=17e, last_copied_object_number=128, snap_seqs={382=18446744073709551614}] 2024-04-24T23:02:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring disabled 2024-04-24T23:02:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-04-24T23:02:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:02:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:02:08.486 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T23:02:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T23:02:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T23:02:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T23:02:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T23:02:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_8 2024-04-24T23:02:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:02:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:02:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T23:02:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-24T23:02:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T23:02:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-04-24T23:02:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T23:02:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T23:02:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T23:02:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-04-24T23:02:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:02:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:02:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T23:02:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T23:02:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T23:02:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T23:02:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T23:02:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_8 2024-04-24T23:02:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:02:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:02:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_8 2024-04-24T23:02:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-04-24T23:02:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T23:02:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-04-24T23:02:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T23:02:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T23:02:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T23:02:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:02:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ purge_snapshots cluster2 mirror image_8 2024-04-24T23:02:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:02:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:02:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T23:02:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2024-04-24T23:02:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_8 2024-04-24T23:02:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:02:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:02:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T23:02:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T23:02:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T23:02:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_8 2024-04-24T23:02:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:02:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:02:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_8 2024-04-24T23:02:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-04-24T23:02:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-04-24T23:02:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-24T23:02:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:02:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T23:02:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_9 2024-04-24T23:02:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_9 512M 2024-04-24T23:02:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:02:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T23:02:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:02:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T23:02:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T23:02:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T23:02:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T23:02:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 512M ']' 2024-04-24T23:02:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=512M 2024-04-24T23:02:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T23:02:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2024-04-24T23:02:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_9 2024-04-24T23:02:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:02:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:02:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T23:02:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T23:02:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-04-24T23:02:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T23:02:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-04-24T23:02:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_9': 2024-04-24T23:02:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T23:02:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T23:02:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T23:02:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 1a4a2db69723 2024-04-24T23:02:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.1a4a2db69723 2024-04-24T23:02:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T23:02:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T23:02:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T23:02:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T23:02:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 23:02:09 2024 2024-04-24T23:02:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 23:02:09 2024 2024-04-24T23:02:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 23:02:09 2024 2024-04-24T23:02:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T23:02:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T23:02:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: be0c452d-a41b-4db3-a767-75065da9316e 2024-04-24T23:02:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T23:02:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_9 2024-04-24T23:02:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:02:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:02:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T23:02:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T23:02:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=12 2024-04-24T23:02:09.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T23:02:09.514 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-04-24T23:02:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 28051456 bytes 2024-04-24T23:02:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 62556160 bytes 2024-04-24T23:02:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 101071872 bytes 2024-04-24T23:02:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 137449472 bytes 2024-04-24T23:02:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 170497024 bytes 2024-04-24T23:02:10.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 188154880 bytes 2024-04-24T23:02:10.590 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 207288320 bytes 2024-04-24T23:02:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 253490176 bytes 2024-04-24T23:02:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 323876864 bytes 2024-04-24T23:02:11.036 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 365879296 bytes 2024-04-24T23:02:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 444066816 bytes 2024-04-24T23:02:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 462469120 bytes 2024-04-24T23:02:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 553266176 bytes 2024-04-24T23:02:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 659972096 bytes 2024-04-24T23:02:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 703040512 bytes 2024-04-24T23:02:11.987 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 735197184 bytes 2024-04-24T23:02:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 763620352 bytes 2024-04-24T23:02:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 821980160 bytes 2024-04-24T23:02:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 869368832 bytes 2024-04-24T23:02:12.534 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 887608320 bytes 2024-04-24T23:02:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 938290176 bytes 2024-04-24T23:02:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 970670080 bytes 2024-04-24T23:02:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 1021560832 bytes 2024-04-24T23:02:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 1049329664 bytes 2024-04-24T23:02:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 1067043840 bytes 2024-04-24T23:02:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 1118058496 bytes 2024-04-24T23:02:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 1137426432 bytes 2024-04-24T23:02:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 1177410560 bytes 2024-04-24T23:02:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1254433792 bytes 2024-04-24T23:02:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1293770752 bytes 2024-04-24T23:02:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1349745664 bytes 2024-04-24T23:02:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1402561536 bytes 2024-04-24T23:02:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1411203072 bytes 2024-04-24T23:02:14.549 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1466930176 bytes 2024-04-24T23:02:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1512769536 bytes 2024-04-24T23:02:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1552004096 bytes 2024-04-24T23:02:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1650797568 bytes 2024-04-24T23:02:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1720977408 bytes 2024-04-24T23:02:15.360 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1787107328 bytes 2024-04-24T23:02:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1816357888 bytes 2024-04-24T23:02:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1865843712 bytes 2024-04-24T23:02:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1944540160 bytes 2024-04-24T23:02:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1976828928 bytes 2024-04-24T23:02:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 2019307520 bytes 2024-04-24T23:02:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 2043620352 bytes 2024-04-24T23:02:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 2067978240 bytes 2024-04-24T23:02:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 2101246976 bytes 2024-04-24T23:02:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 2149937152 bytes 2024-04-24T23:02:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 2188090368 bytes 2024-04-24T23:02:16.873 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 2211571712 bytes 2024-04-24T23:02:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 2299523072 bytes 2024-04-24T23:02:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 2308377600 bytes 2024-04-24T23:02:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 2362372096 bytes 2024-04-24T23:02:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2377761792 bytes 2024-04-24T23:02:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2418399232 bytes 2024-04-24T23:02:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2474586112 bytes 2024-04-24T23:02:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2527201280 bytes 2024-04-24T23:02:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2566506496 bytes 2024-04-24T23:02:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2579064832 bytes 2024-04-24T23:02:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2616838144 bytes 2024-04-24T23:02:18.396 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2631479296 bytes 2024-04-24T23:02:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2663560192 bytes 2024-04-24T23:02:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2702066688 bytes 2024-04-24T23:02:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2742035456 bytes 2024-04-24T23:02:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2754985984 bytes 2024-04-24T23:02:19.058 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2775623680 bytes 2024-04-24T23:02:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2810955776 bytes 2024-04-24T23:02:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2903628800 bytes 2024-04-24T23:02:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2960772096 bytes 2024-04-24T23:02:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 2961406976 bytes 2024-04-24T23:02:19.801 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 2989179904 bytes 2024-04-24T23:02:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 3067218944 bytes 2024-04-24T23:02:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 3115512832 bytes 2024-04-24T23:02:20.253 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 3133566976 bytes 2024-04-24T23:02:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 3153166336 bytes 2024-04-24T23:02:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 3154048000 bytes 2024-04-24T23:02:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 3196753920 bytes 2024-04-24T23:02:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 3272960000 bytes 2024-04-24T23:02:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7900 IOs, 3305927680 bytes 2024-04-24T23:02:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8000 IOs, 3340939264 bytes 2024-04-24T23:02:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8100 IOs, 3371141120 bytes 2024-04-24T23:02:21.416 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8200 IOs, 3459329024 bytes 2024-04-24T23:02:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T23:02:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T23:02:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T23:02:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:02:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2024-04-24T23:02:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:02:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:02:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T23:02:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=present 2024-04-24T23:02:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T23:02:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T23:02:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T23:02:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_9 2024-04-24T23:02:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:02:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:02:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_9 2024-04-24T23:02:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-24T23:02:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T23:02:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id=161635afbd03 2024-04-24T23:02:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T23:02:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T23:02:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_9 present 161635afbd03 2024-04-24T23:02:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:02:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:02:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T23:02:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=present 2024-04-24T23:02:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id=161635afbd03 2024-04-24T23:02:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T23:02:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T23:02:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_9 2024-04-24T23:02:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:02:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:02:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_9 2024-04-24T23:02:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-24T23:02:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T23:02:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id=161635afbd03 2024-04-24T23:02:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 161635afbd03 2024-04-24T23:02:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -z 161635afbd03 -o 161635afbd03 = 161635afbd03 2024-04-24T23:02:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=present 2024-04-24T23:02:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test present = present 2024-04-24T23:02:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:02:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ disable_mirror cluster2 mirror image_9 2024-04-24T23:02:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:02:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:02:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T23:02:21.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2024-04-24T23:02:23.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:02:23.833+0000 7f7746536700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c81fe87d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T23:02:23.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:02:23.833+0000 7f774cd43700 -1 rbd::mirror::ImageReplayer: 0x55c81ffd4000 [3/be0c452d-a41b-4db3-a767-75065da9316e] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T23:02:50.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:02:50.639+0000 7f572a7fb700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55db40b22000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=384 2024-04-24T23:02:50.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:02:50.639+0000 7f572a7fb700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55db40b22000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=384, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=d9b73256-b8ac-4751-9b60-95a0234fa37c, primary_snap_id=180, last_copied_object_number=128, snap_seqs={384=18446744073709551614}] 2024-04-24T23:02:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring disabled 2024-04-24T23:02:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-04-24T23:02:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:02:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:02:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T23:02:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T23:02:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T23:02:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T23:02:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T23:02:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_9 2024-04-24T23:02:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:02:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:02:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_9 2024-04-24T23:02:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-24T23:02:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T23:02:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-04-24T23:02:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T23:02:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T23:02:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T23:02:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-04-24T23:02:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:02:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:02:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T23:02:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T23:02:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T23:02:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T23:02:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T23:02:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_9 2024-04-24T23:02:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:02:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:02:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_9 2024-04-24T23:02:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-04-24T23:02:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T23:02:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-04-24T23:02:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T23:02:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T23:02:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T23:02:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:02:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ purge_snapshots cluster2 mirror image_9 2024-04-24T23:02:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:02:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:02:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T23:02:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2024-04-24T23:02:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_9 2024-04-24T23:02:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:02:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:02:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T23:02:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T23:02:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T23:02:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_9 2024-04-24T23:02:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:02:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:02:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_9 2024-04-24T23:02:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-04-24T23:02:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-04-24T23:02:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-24T23:02:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:02:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for i in `seq 1 10` 2024-04-24T23:02:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image=image_10 2024-04-24T23:02:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ create_image cluster2 mirror image_10 512M 2024-04-24T23:02:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:02:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T23:02:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:02:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T23:02:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T23:02:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T23:02:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local size=128 2024-04-24T23:02:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -n 512M ']' 2024-04-24T23:02:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ size=512M 2024-04-24T23:02:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ shift 2024-04-24T23:02:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2024-04-24T23:02:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ enable_mirror cluster2 mirror image_10 2024-04-24T23:02:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:02:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:02:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T23:02:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local mode=snapshot 2024-04-24T23:02:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-04-24T23:02:51.645 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring enabled 2024-04-24T23:02:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-04-24T23:02:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:rbd image 'image_10': 2024-04-24T23:02:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: size 512 MiB in 128 objects 2024-04-24T23:02:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: order 22 (4 MiB objects) 2024-04-24T23:02:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: snapshot_count: 1 2024-04-24T23:02:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: id: 1a625d57f9ba 2024-04-24T23:02:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: block_name_prefix: rbd_data.1a625d57f9ba 2024-04-24T23:02:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: format: 2 2024-04-24T23:02:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T23:02:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: op_features: 2024-04-24T23:02:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: flags: 2024-04-24T23:02:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: create_timestamp: Wed Apr 24 23:02:51 2024 2024-04-24T23:02:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: access_timestamp: Wed Apr 24 23:02:51 2024 2024-04-24T23:02:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: modify_timestamp: Wed Apr 24 23:02:51 2024 2024-04-24T23:02:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring state: enabled 2024-04-24T23:02:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring mode: snapshot 2024-04-24T23:02:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring global id: a44a9c86-73ab-4c4f-9bfd-ea25e12500bc 2024-04-24T23:02:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout: mirroring primary: true 2024-04-24T23:02:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stress_write_image cluster2 mirror image_10 2024-04-24T23:02:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:02:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:02:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T23:02:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-04-24T23:02:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local duration=12 2024-04-24T23:02:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T23:02:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-04-24T23:02:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:100 IOs, 28342272 bytes 2024-04-24T23:02:52.135 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:200 IOs, 90697728 bytes 2024-04-24T23:02:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:300 IOs, 95358976 bytes 2024-04-24T23:02:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:400 IOs, 132419584 bytes 2024-04-24T23:02:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:500 IOs, 162495488 bytes 2024-04-24T23:02:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:600 IOs, 205976576 bytes 2024-04-24T23:02:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:700 IOs, 265667584 bytes 2024-04-24T23:02:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:800 IOs, 289111040 bytes 2024-04-24T23:02:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:900 IOs, 289712128 bytes 2024-04-24T23:02:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1000 IOs, 313423872 bytes 2024-04-24T23:02:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1100 IOs, 352033792 bytes 2024-04-24T23:02:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1200 IOs, 376936448 bytes 2024-04-24T23:02:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1300 IOs, 387052544 bytes 2024-04-24T23:02:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1400 IOs, 468832256 bytes 2024-04-24T23:02:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1500 IOs, 491783168 bytes 2024-04-24T23:02:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1600 IOs, 492392448 bytes 2024-04-24T23:02:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1700 IOs, 531700736 bytes 2024-04-24T23:02:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1800 IOs, 545901568 bytes 2024-04-24T23:02:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:1900 IOs, 580064256 bytes 2024-04-24T23:02:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2000 IOs, 609360896 bytes 2024-04-24T23:02:54.679 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2100 IOs, 658833408 bytes 2024-04-24T23:02:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2200 IOs, 699157504 bytes 2024-04-24T23:02:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2300 IOs, 759892992 bytes 2024-04-24T23:02:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2400 IOs, 791289856 bytes 2024-04-24T23:02:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2500 IOs, 841198592 bytes 2024-04-24T23:02:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2600 IOs, 846363648 bytes 2024-04-24T23:02:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2700 IOs, 922993664 bytes 2024-04-24T23:02:55.721 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2800 IOs, 985538560 bytes 2024-04-24T23:02:55.902 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:2900 IOs, 1062654976 bytes 2024-04-24T23:02:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3000 IOs, 1091005440 bytes 2024-04-24T23:02:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3100 IOs, 1125070848 bytes 2024-04-24T23:02:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3200 IOs, 1177303040 bytes 2024-04-24T23:02:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3300 IOs, 1226535936 bytes 2024-04-24T23:02:56.616 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3400 IOs, 1271529472 bytes 2024-04-24T23:02:56.740 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3500 IOs, 1295004672 bytes 2024-04-24T23:02:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3600 IOs, 1359026176 bytes 2024-04-24T23:02:57.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3700 IOs, 1361957888 bytes 2024-04-24T23:02:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3800 IOs, 1375840256 bytes 2024-04-24T23:02:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:3900 IOs, 1420263424 bytes 2024-04-24T23:02:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4000 IOs, 1459693568 bytes 2024-04-24T23:02:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4100 IOs, 1516469248 bytes 2024-04-24T23:02:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4200 IOs, 1530976256 bytes 2024-04-24T23:02:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4300 IOs, 1553043456 bytes 2024-04-24T23:02:57.960 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4400 IOs, 1589166080 bytes 2024-04-24T23:02:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4500 IOs, 1649255424 bytes 2024-04-24T23:02:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4600 IOs, 1652523008 bytes 2024-04-24T23:02:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4700 IOs, 1673149440 bytes 2024-04-24T23:02:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4800 IOs, 1673769984 bytes 2024-04-24T23:02:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:4900 IOs, 1711709184 bytes 2024-04-24T23:02:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5000 IOs, 1844329472 bytes 2024-04-24T23:02:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5100 IOs, 1884618752 bytes 2024-04-24T23:02:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5200 IOs, 1921169408 bytes 2024-04-24T23:02:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5300 IOs, 1964647424 bytes 2024-04-24T23:02:59.392 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5400 IOs, 2014049280 bytes 2024-04-24T23:02:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5500 IOs, 2036052992 bytes 2024-04-24T23:02:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5600 IOs, 2052824064 bytes 2024-04-24T23:02:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5700 IOs, 2088017920 bytes 2024-04-24T23:02:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5800 IOs, 2148474880 bytes 2024-04-24T23:03:00.086 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:5900 IOs, 2198301696 bytes 2024-04-24T23:03:00.218 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6000 IOs, 2229480448 bytes 2024-04-24T23:03:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6100 IOs, 2266078208 bytes 2024-04-24T23:03:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6200 IOs, 2286049280 bytes 2024-04-24T23:03:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6300 IOs, 2324229120 bytes 2024-04-24T23:03:00.709 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6400 IOs, 2324815872 bytes 2024-04-24T23:03:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6500 IOs, 2368494592 bytes 2024-04-24T23:03:01.019 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6600 IOs, 2432947200 bytes 2024-04-24T23:03:01.233 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6700 IOs, 2543029248 bytes 2024-04-24T23:03:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6800 IOs, 2548689920 bytes 2024-04-24T23:03:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:6900 IOs, 2553096192 bytes 2024-04-24T23:03:01.609 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7000 IOs, 2625341440 bytes 2024-04-24T23:03:01.795 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7100 IOs, 2708841472 bytes 2024-04-24T23:03:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7200 IOs, 2709447680 bytes 2024-04-24T23:03:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7300 IOs, 2756855808 bytes 2024-04-24T23:03:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7400 IOs, 2776152064 bytes 2024-04-24T23:03:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7500 IOs, 2863512576 bytes 2024-04-24T23:03:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7600 IOs, 2896396288 bytes 2024-04-24T23:03:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7700 IOs, 2917713920 bytes 2024-04-24T23:03:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7800 IOs, 2969948160 bytes 2024-04-24T23:03:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:7900 IOs, 2975793152 bytes 2024-04-24T23:03:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8000 IOs, 3008601088 bytes 2024-04-24T23:03:03.317 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8100 IOs, 3033636864 bytes 2024-04-24T23:03:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8200 IOs, 3082341376 bytes 2024-04-24T23:03:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:8300 IOs, 3117276160 bytes 2024-04-24T23:03:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=124 2024-04-24T23:03:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T23:03:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 124 -eq 124 ']' 2024-04-24T23:03:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:03:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2024-04-24T23:03:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:03:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:03:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T23:03:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=present 2024-04-24T23:03:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T23:03:03.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T23:03:03.704 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T23:03:03.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_10 2024-04-24T23:03:03.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:03:03.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:03:03.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T23:03:03.706 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T23:03:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-24T23:03:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id=16161652f1cb 2024-04-24T23:03:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T23:03:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T23:03:03.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_10 present 16161652f1cb 2024-04-24T23:03:03.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:03:03.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:03:03.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T23:03:03.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=present 2024-04-24T23:03:03.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id=16161652f1cb 2024-04-24T23:03:03.848 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T23:03:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T23:03:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_10 2024-04-24T23:03:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:03:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:03:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T23:03:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-24T23:03:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T23:03:03.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id=16161652f1cb 2024-04-24T23:03:03.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 16161652f1cb 2024-04-24T23:03:03.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -z 16161652f1cb -o 16161652f1cb = 16161652f1cb 2024-04-24T23:03:03.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_state=present 2024-04-24T23:03:03.886 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test present = present 2024-04-24T23:03:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:03:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ disable_mirror cluster2 mirror image_10 2024-04-24T23:03:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:03:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:03:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T23:03:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2024-04-24T23:03:11.578 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.965722891553044) > weight (0.25) 2024-04-24T23:03:11.578 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-04-24T23:03:11.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-04-24T23:03:11.579 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-04-24T23:03:11.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:11.578+0000 7f774d544700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 81416) UID: 1000 2024-04-24T23:03:11.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T23:03:15.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:15.637+0000 7f3226dcd700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5583f02b3a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T23:03:15.639 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:15.637+0000 7f322d5da700 -1 rbd::mirror::ImageReplayer: 0x5583ed11f400 [3/a44a9c86-73ab-4c4f-9bfd-ea25e12500bc] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T23:03:16.920 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4011394168482796) > weight (0.25) 2024-04-24T23:03:16.920 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-04-24T23:03:16.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-04-24T23:03:16.920 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-04-24T23:03:16.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:16.920+0000 7f648efcd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 82557) UID: 1000 2024-04-24T23:03:17.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T23:03:18.579 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-04-24T23:03:18.580 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T23:03:18.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:03:18.580 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-04-24T23:03:18.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-04-24T23:03:18.581 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-04-24T23:03:18.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-04-24T23:03:18.584 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2024-04-24T23:03:18.921 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-04-24T23:03:18.921 INFO:teuthology.orchestra.run:waiting for 600 2024-04-24T23:03:18.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:03:18.921 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-04-24T23:03:18.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-04-24T23:03:18.922 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-04-24T23:03:18.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-04-24T23:03:18.925 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2024-04-24T23:03:23.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:23.688+0000 7f32275ce700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5583f0383860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-24T23:03:23.689 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:23.688+0000 7f322dddb700 -1 rbd::mirror::ImageReplayer: 0x5583ed11f400 [3/a44a9c86-73ab-4c4f-9bfd-ea25e12500bc] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-24T23:03:34.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:34.475+0000 7f5729ffa700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55db40d8b400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=386 2024-04-24T23:03:34.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:34.475+0000 7f5729ffa700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55db40d8b400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=386, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=d9b73256-b8ac-4751-9b60-95a0234fa37c, primary_snap_id=182, last_copied_object_number=128, snap_seqs={386=18446744073709551614}] 2024-04-24T23:03:34.491 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:Mirroring disabled 2024-04-24T23:03:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-04-24T23:03:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:03:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:03:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T23:03:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local state=deleted 2024-04-24T23:03:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T23:03:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local s 2024-04-24T23:03:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T23:03:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_10 2024-04-24T23:03:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:03:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:03:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T23:03:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-24T23:03:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T23:03:34.537 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-04-24T23:03:34.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ image_id= 2024-04-24T23:03:34.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-04-24T23:03:34.541 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0.1 2024-04-24T23:03:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-04-24T23:03:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:03:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:03:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T23:03:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local test_state=deleted 2024-04-24T23:03:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image_id= 2024-04-24T23:03:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_state=deleted 2024-04-24T23:03:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local current_image_id 2024-04-24T23:03:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ get_image_id cluster1 mirror image_10 2024-04-24T23:03:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:03:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:03:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local image=image_10 2024-04-24T23:03:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-04-24T23:03:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-04-24T23:03:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-04-24T23:03:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ current_image_id= 2024-04-24T23:03:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n '' 2024-04-24T23:03:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test deleted = deleted 2024-04-24T23:03:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:03:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ purge_snapshots cluster2 mirror image_10 2024-04-24T23:03:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:03:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:03:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T23:03:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2024-04-24T23:03:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image_retry cluster2 mirror image_10 2024-04-24T23:03:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:03:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:03:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T23:03:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 4 8 16 32 2024-04-24T23:03:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T23:03:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ remove_image cluster2 mirror image_10 2024-04-24T23:03:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:03:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:03:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local image=image_10 2024-04-24T23:03:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-04-24T23:03:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-04-24T23:03:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-24T23:03:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:03:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2024-04-24T23:03:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ tee -a /tmp/tmp.DCvHjwkiaM/rbd-mirror.test.log 2024-04-24T23:03:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ date '+%F %T' 2024-04-24T23:03:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo 2024-04-24 23:03:35 TEST: check if removed 'images'\''' OMAP are removed 2024-04-24T23:03:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:2024-04-24 23:03:35 TEST: check if removed images' OMAP are removed 2024-04-24T23:03:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-04-24T23:03:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:03:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:03:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-04-24T23:03:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:03:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:03:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local obj_name=rbd_mirroring 2024-04-24T23:03:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local filter=status_global 2024-04-24T23:03:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-24T23:03:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T23:03:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T23:03:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-04-24T23:03:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:03:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:03:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirroring 2024-04-24T23:03:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local filter=status_global 2024-04-24T23:03:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-04-24T23:03:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:03:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -c status_global 2024-04-24T23:03:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:03:35.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirroring 2024-04-24T23:03:35.045 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-04-24T23:03:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 0 = 0 2024-04-24T23:03:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=0 2024-04-24T23:03:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T23:03:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -eq 0 ']' 2024-04-24T23:03:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:03:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-04-24T23:03:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:03:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:03:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local obj_name=rbd_mirroring 2024-04-24T23:03:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local filter=image_ 2024-04-24T23:03:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-24T23:03:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T23:03:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T23:03:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-04-24T23:03:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:03:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:03:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirroring 2024-04-24T23:03:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local filter=image_ 2024-04-24T23:03:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-04-24T23:03:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:03:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:03:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirroring 2024-04-24T23:03:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-04-24T23:03:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -c image_ 2024-04-24T23:03:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 0 = 0 2024-04-24T23:03:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=0 2024-04-24T23:03:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T23:03:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -eq 0 ']' 2024-04-24T23:03:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:03:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-04-24T23:03:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:03:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:03:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local obj_name=rbd_mirror_leader 2024-04-24T23:03:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local filter=image_map 2024-04-24T23:03:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-24T23:03:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T23:03:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T23:03:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-04-24T23:03:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:03:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:03:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirror_leader 2024-04-24T23:03:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local filter=image_map 2024-04-24T23:03:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-04-24T23:03:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:03:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:03:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirror_leader 2024-04-24T23:03:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-04-24T23:03:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -c image_map 2024-04-24T23:03:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 1 = 0 2024-04-24T23:03:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=1 2024-04-24T23:03:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T23:03:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 -eq 0 ']' 2024-04-24T23:03:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-24T23:03:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 1 2024-04-24T23:03:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T23:03:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-04-24T23:03:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:03:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:03:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirror_leader 2024-04-24T23:03:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local filter=image_map 2024-04-24T23:03:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-04-24T23:03:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:03:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:03:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirror_leader 2024-04-24T23:03:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-04-24T23:03:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -c image_map 2024-04-24T23:03:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 1 = 0 2024-04-24T23:03:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=1 2024-04-24T23:03:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T23:03:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 1 -eq 0 ']' 2024-04-24T23:03:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-24T23:03:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 2 2024-04-24T23:03:38.168 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T23:03:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-04-24T23:03:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:03:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:03:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirror_leader 2024-04-24T23:03:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local filter=image_map 2024-04-24T23:03:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-04-24T23:03:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster1 2024-04-24T23:03:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:03:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirror_leader 2024-04-24T23:03:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-04-24T23:03:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -c image_map 2024-04-24T23:03:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 0 = 0 2024-04-24T23:03:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=0 2024-04-24T23:03:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T23:03:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -eq 0 ']' 2024-04-24T23:03:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:03:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-04-24T23:03:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:03:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:03:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-04-24T23:03:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:03:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:03:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local obj_name=rbd_mirroring 2024-04-24T23:03:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local filter=status_global 2024-04-24T23:03:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-24T23:03:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T23:03:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T23:03:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-04-24T23:03:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T23:03:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:03:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirroring 2024-04-24T23:03:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local filter=status_global 2024-04-24T23:03:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-04-24T23:03:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T23:03:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:03:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirroring 2024-04-24T23:03:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -c status_global 2024-04-24T23:03:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-04-24T23:03:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 0 = 0 2024-04-24T23:03:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=0 2024-04-24T23:03:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T23:03:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -eq 0 ']' 2024-04-24T23:03:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:03:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-04-24T23:03:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:03:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:03:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local obj_name=rbd_mirroring 2024-04-24T23:03:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local filter=image_ 2024-04-24T23:03:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-24T23:03:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T23:03:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T23:03:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-04-24T23:03:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T23:03:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:03:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirroring 2024-04-24T23:03:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local filter=image_ 2024-04-24T23:03:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-04-24T23:03:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T23:03:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:03:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirroring 2024-04-24T23:03:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-04-24T23:03:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -c image_ 2024-04-24T23:03:38.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 0 = 0 2024-04-24T23:03:38.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=0 2024-04-24T23:03:38.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T23:03:38.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -eq 0 ']' 2024-04-24T23:03:38.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:03:38.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-04-24T23:03:38.267 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:03:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local pool=mirror 2024-04-24T23:03:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local obj_name=rbd_mirror_leader 2024-04-24T23:03:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local filter=image_map 2024-04-24T23:03:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-04-24T23:03:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ sleep 0 2024-04-24T23:03:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T23:03:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-04-24T23:03:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T23:03:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:03:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirror_leader 2024-04-24T23:03:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local filter=image_map 2024-04-24T23:03:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-04-24T23:03:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local cluster=cluster2 2024-04-24T23:03:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local pool=mirror 2024-04-24T23:03:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ local obj_name=rbd_mirror_leader 2024-04-24T23:03:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-04-24T23:03:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ grep -c image_map 2024-04-24T23:03:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test 0 = 0 2024-04-24T23:03:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ error_code=0 2024-04-24T23:03:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set -e 2024-04-24T23:03:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -eq 0 ']' 2024-04-24T23:03:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 0 2024-04-24T23:03:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ cleanup 0 2024-04-24T23:03:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local error_code=0 2024-04-24T23:03:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ set +e 2024-04-24T23:03:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -ne 0 ']' 2024-04-24T23:03:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -z '' ']' 2024-04-24T23:03:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster instance 2024-04-24T23:03:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T23:03:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-04-24T23:03:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:pool 'mirror' does not exist 2024-04-24T23:03:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T23:03:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-04-24T23:03:39.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:39.876+0000 7f5731008700 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55db3e39d8c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-24T23:03:39.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:39.876+0000 7f5731008700 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55db408af200 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-24T23:03:39.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:39.876+0000 7f5731008700 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55db3f04a480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-24T23:03:39.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:39.876+0000 7f5731008700 -1 rbd::mirror::PoolWatcher: 0x55db3c9d1680 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T23:03:39.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:39.881+0000 7f5737815700 -1 rbd::mirror::LeaderWatcher: 0x55db3ef18c00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-04-24T23:03:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:pool 'mirror' does not exist 2024-04-24T23:03:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T23:03:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-04-24T23:03:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:pool 'mirror_parent' does not exist 2024-04-24T23:03:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ CEPH_ARGS= 2024-04-24T23:03:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-04-24T23:03:41.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:41.628+0000 7f3226dcd700 -1 rbd::mirror::InstanceWatcher: 0x5583f02ee708 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:41.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:41.656+0000 7f322dddb700 -1 rbd::mirror::LeaderWatcher: 0x5583ee904c00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-04-24T23:03:41.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:41.656+0000 7f3226dcd700 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5583f02ad200 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-24T23:03:41.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:41.656+0000 7f3226dcd700 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5583eea2c480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-24T23:03:41.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:41.656+0000 7f3226dcd700 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5583eea2c6c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-24T23:03:41.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:41.657+0000 7f3226dcd700 -1 rbd::mirror::PoolWatcher: 0x5583ec3c05a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T23:03:41.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:41.657+0000 7f32275ce700 -1 rbd::mirror::PoolWatcher: 0x5583ec3c0000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T23:03:41.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:41.657+0000 7f32275ce700 -1 rbd::mirror::PoolWatcher: 0x5583ec3c03c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T23:03:41.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:41.657+0000 7f32205c0700 -1 rbd::mirror::PoolWatcher: 0x5583ec3c12c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T23:03:41.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:41.657+0000 7f32205c0700 -1 rbd::mirror::PoolWatcher: 0x5583ec3c10e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T23:03:41.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:41.657+0000 7f32205c0700 -1 rbd::mirror::PoolWatcher: 0x5583ec3c14a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T23:03:41.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:41.658+0000 7f322dddb700 -1 rbd::mirror::LeaderWatcher: 0x5583ee904c00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-04-24T23:03:41.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:41.658+0000 7f322dddb700 -1 rbd::mirror::LeaderWatcher: 0x5583ee904c00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-04-24T23:03:42.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.068+0000 7f16d57d1700 -1 rbd::mirror::RemotePoolPoller: 0x561c6eb0d040 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-04-24T23:03:42.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.068+0000 7f16d4fd0700 -1 rbd::mirror::RemotePoolPoller: 0x561c6eb0d040 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-04-24T23:03:42.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.068+0000 7f16d4fd0700 -1 rbd::mirror::PoolReplayer: 0x561c6ec4db00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-04-24T23:03:42.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.069+0000 7f16db7dd700 -1 rbd::mirror::InstanceWatcher: 0x561c6eb33500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.069+0000 7f16db7dd700 -1 rbd::mirror::InstanceWatcher: 0x561c72a3c000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.071+0000 7f16d4fd0700 -1 rbd::mirror::MirrorStatusUpdater 0x561c6eb9bc20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.071+0000 7f16d4fd0700 -1 rbd::mirror::MirrorStatusUpdater 0x561c71c5e360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.071+0000 7f16e27eb700 -1 rbd::mirror::NamespaceReplayer: 0x561c711849c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.071+0000 7f16e27eb700 -1 rbd::mirror::NamespaceReplayer: 0x561c71184b60 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.071+0000 7f16db7dd700 -1 rbd::mirror::MirrorStatusUpdater 0x561c71c5e000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.071+0000 7f16e1fea700 -1 rbd::mirror::NamespaceReplayer: 0x561c711849c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.114+0000 7f16c77b5700 -1 rbd::mirror::RemotePoolPoller: 0x561c6eb0d6c0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-04-24T23:03:42.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.114+0000 7f16c77b5700 -1 rbd::mirror::RemotePoolPoller: 0x561c6eb0d6c0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-04-24T23:03:42.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.114+0000 7f16c77b5700 -1 rbd::mirror::PoolReplayer: 0x561c71050f00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-04-24T23:03:42.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.162+0000 7f16e1fea700 -1 rbd::mirror::LeaderWatcher: 0x561c71050c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T23:03:42.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.162+0000 7f16dbfde700 -1 rbd::mirror::InstanceWatcher: 0x561c6eb33180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.163+0000 7f16d57d1700 -1 rbd::mirror::MirrorStatusUpdater 0x561c71164240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.163+0000 7f16e27eb700 -1 rbd::mirror::NamespaceReplayer: 0x561c71184000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.163+0000 7f16dbfde700 -1 rbd::mirror::MirrorStatusUpdater 0x561c70527440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.163+0000 7f16e1fea700 -1 rbd::mirror::NamespaceReplayer: 0x561c71184000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.165+0000 7f16ce7c3700 -1 rbd::mirror::InstanceWatcher: 0x561c6eb33880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.165+0000 7f16c7fb6700 -1 rbd::mirror::MirrorStatusUpdater 0x561c6eb9be60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.165+0000 7f16e27eb700 -1 rbd::mirror::NamespaceReplayer: 0x561c71184820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:pool 'mirror_parent' does not exist 2024-04-24T23:03:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-04-24T23:03:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stop_mirrors cluster1 2024-04-24T23:03:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1 2024-04-24T23:03:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local sig= 2024-04-24T23:03:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T23:03:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T23:03:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stop_mirror cluster1:0 '' 2024-04-24T23:03:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:0 2024-04-24T23:03:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local sig= 2024-04-24T23:03:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T23:03:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T23:03:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T23:03:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stop_mirror cluster1:1 '' 2024-04-24T23:03:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:1 2024-04-24T23:03:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local sig= 2024-04-24T23:03:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T23:03:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T23:03:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T23:03:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stop_mirror cluster1:2 '' 2024-04-24T23:03:42.192 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:2 2024-04-24T23:03:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local sig= 2024-04-24T23:03:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T23:03:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T23:03:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T23:03:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stop_mirror cluster1:3 '' 2024-04-24T23:03:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster1:3 2024-04-24T23:03:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local sig= 2024-04-24T23:03:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T23:03:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T23:03:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-04-24T23:03:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stop_mirrors cluster2 2024-04-24T23:03:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2 2024-04-24T23:03:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local sig= 2024-04-24T23:03:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:++ seq 0 3 2024-04-24T23:03:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T23:03:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stop_mirror cluster2:0 '' 2024-04-24T23:03:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2:0 2024-04-24T23:03:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local sig= 2024-04-24T23:03:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T23:03:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T23:03:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T23:03:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stop_mirror cluster2:1 '' 2024-04-24T23:03:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2:1 2024-04-24T23:03:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local sig= 2024-04-24T23:03:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T23:03:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T23:03:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T23:03:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stop_mirror cluster2:2 '' 2024-04-24T23:03:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2:2 2024-04-24T23:03:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local sig= 2024-04-24T23:03:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T23:03:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T23:03:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T23:03:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ stop_mirror cluster2:3 '' 2024-04-24T23:03:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local cluster=cluster2:3 2024-04-24T23:03:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ local sig= 2024-04-24T23:03:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test -n 1 2024-04-24T23:03:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ return 2024-04-24T23:03:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' -z 1 ']' 2024-04-24T23:03:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ test '' = /tmp/tmp.DCvHjwkiaM 2024-04-24T23:03:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ rm -Rf /tmp/tmp.DCvHjwkiaM 2024-04-24T23:03:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ '[' 0 -eq 0 ']' 2024-04-24T23:03:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ echo OK 2024-04-24T23:03:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stdout:OK 2024-04-24T23:03:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi134.stderr:+ exit 0 2024-04-24T23:03:42.198 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-24T23:03:42.198 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-04-24T23:03:42.268 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-04-24T23:03:42.268 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-04-24T23:03:42.572 DEBUG:teuthology.parallel:result is None 2024-04-24T23:03:42.572 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-04-24T23:03:42.600 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-04-24T23:03:42.600 DEBUG:teuthology.orchestra.run.smithi134:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-24T23:03:42.674 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-04-24T23:03:42.675 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-04-24T23:03:42.688 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-04-24T23:03:42.688 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 18 2024-04-24T23:03:42.688 INFO:tasks.rbd_mirror_thrash:done joining 2024-04-24T23:03:42.688 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-04-24T23:03:42.699 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-04-24T23:03:42.699 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 20 2024-04-24T23:03:42.699 INFO:tasks.rbd_mirror_thrash:done joining 2024-04-24T23:03:42.699 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-24T23:03:42.712 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2024-04-24T23:03:42.713 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:03:42.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.714+0000 7f8f012e9700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 83297) UID: 1000 2024-04-24T23:03:42.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.714+0000 7f8eecac0700 -1 rbd::mirror::image_deleter::TrashWatcher: 0x56445826c480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-24T23:03:42.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.715+0000 7f8f00ae8700 -1 rbd::mirror::InstanceWatcher: 0x564455c17500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6662 object: (2) No such file or directory 2024-04-24T23:03:42.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.715+0000 7f8f00ae8700 -1 rbd::mirror::InstanceWatcher: 0x564455c17c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6662 object: (2) No such file or directory 2024-04-24T23:03:42.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.715+0000 7f8eecac0700 -1 rbd::mirror::PoolWatcher: 0x564455bec5a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T23:03:42.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.715+0000 7f8ef9ada700 -1 rbd::mirror::InstanceWatcher: 0x564455c17500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.715+0000 7f8ef9ada700 -1 rbd::mirror::InstanceWatcher: 0x564455c17c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.715+0000 7f8ee5ab2700 -1 rbd::mirror::PoolWatcher: 0x564455bec3c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T23:03:42.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.715+0000 7f8efa2db700 -1 rbd::mirror::MirrorStatusUpdater 0x5644582d0000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.715+0000 7f8f002e7700 -1 rbd::mirror::NamespaceReplayer: 0x5644582ce000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.715+0000 7f8ef3ace700 -1 rbd::mirror::MirrorStatusUpdater 0x5644582d0480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.715+0000 7f8f002e7700 -1 rbd::mirror::NamespaceReplayer: 0x5644582ce1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.715+0000 7f8f00ae8700 -1 rbd::mirror::LeaderWatcher: 0x564459a18900 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-04-24T23:03:42.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.715+0000 7f8f00ae8700 -1 rbd::mirror::LeaderWatcher: 0x564459a18900 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-04-24T23:03:42.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.715+0000 7f8efa2db700 -1 rbd::mirror::MirrorStatusUpdater 0x5644582d0120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.715+0000 7f8f002e7700 -1 rbd::mirror::NamespaceReplayer: 0x5644582ce1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.715+0000 7f8f00ae8700 -1 rbd::mirror::LeaderWatcher: 0x564459a18900 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T23:03:42.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.716+0000 7f8f00ae8700 -1 rbd::mirror::InstanceWatcher: 0x564459ad8000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6664 object: (2) No such file or directory 2024-04-24T23:03:42.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.716+0000 7f8eecac0700 -1 rbd::mirror::InstanceWatcher: 0x564459ad8000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.716+0000 7f8ee62b3700 -1 rbd::mirror::MirrorStatusUpdater 0x564455c7fe60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.716+0000 7f8f00ae8700 -1 rbd::mirror::NamespaceReplayer: 0x564459ad2000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.716+0000 7f8eec2bf700 -1 rbd::mirror::MirrorStatusUpdater 0x564458e96240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.716+0000 7f8f002e7700 -1 rbd::mirror::NamespaceReplayer: 0x564459ad2000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.718+0000 7f8f00ae8700 -1 rbd::mirror::LeaderWatcher: 0x5644581acf00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T23:03:42.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.718+0000 7f8f00ae8700 -1 rbd::mirror::InstanceWatcher: 0x564455c17180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6662 object: (2) No such file or directory 2024-04-24T23:03:42.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.719+0000 7f8efa2db700 -1 rbd::mirror::InstanceWatcher: 0x564455c17180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.719+0000 7f8ef9ada700 -1 rbd::mirror::MirrorStatusUpdater 0x564457631440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.719+0000 7f8f00ae8700 -1 rbd::mirror::NamespaceReplayer: 0x5644581944e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T23:03:42.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:03:42.731 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3' 2024-04-24T23:03:42.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2024-04-24T23:03:42.732 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-24T23:03:42.742 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2024-04-24T23:03:42.742 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:03:42.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.770+0000 7f322e5dc700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 76255) UID: 1000 2024-04-24T23:03:42.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.771+0000 7f322d5da700 -1 rbd::mirror::LeaderWatcher: 0x5583ef589b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T23:03:42.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.771+0000 7f322dddb700 -1 rbd::mirror::InstanceWatcher: 0x5583ec3eb500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6194 object: (2) No such file or directory 2024-04-24T23:03:42.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.771+0000 7f322dddb700 -1 rbd::mirror::InstanceWatcher: 0x5583ec3eb880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6194 object: (2) No such file or directory 2024-04-24T23:03:42.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.771+0000 7f322dddb700 -1 rbd::mirror::InstanceWatcher: 0x5583ec3ebc00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6197 object: (2) No such file or directory 2024-04-24T23:03:42.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.771+0000 7f32275ce700 -1 rbd::mirror::InstanceWatcher: 0x5583ec3eb500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.771+0000 7f3226dcd700 -1 rbd::mirror::InstanceWatcher: 0x5583ec3eb880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.771+0000 7f3219db3700 -1 rbd::mirror::InstanceWatcher: 0x5583ec3ebc00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.772+0000 7f32205c0700 -1 rbd::mirror::MirrorStatusUpdater 0x5583eea7c360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.772+0000 7f322d5da700 -1 rbd::mirror::NamespaceReplayer: 0x5583eea78000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.772+0000 7f3220dc1700 -1 rbd::mirror::MirrorStatusUpdater 0x5583eea7c480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.772+0000 7f322d5da700 -1 rbd::mirror::NamespaceReplayer: 0x5583eea781a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.772+0000 7f32275ce700 -1 rbd::mirror::MirrorStatusUpdater 0x5583eea7c000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.775 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.772+0000 7f32275ce700 -1 rbd::mirror::MirrorStatusUpdater 0x5583eea7c120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.772+0000 7f322dddb700 -1 rbd::mirror::NamespaceReplayer: 0x5583eea78000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.772+0000 7f322dddb700 -1 rbd::mirror::NamespaceReplayer: 0x5583eea781a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.772+0000 7f3212da5700 -1 rbd::mirror::MirrorStatusUpdater 0x5583f01fc5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.772+0000 7f322d5da700 -1 rbd::mirror::NamespaceReplayer: 0x5583f02b2000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.772+0000 7f32195b2700 -1 rbd::mirror::MirrorStatusUpdater 0x5583ef66fe60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.772+0000 7f322dddb700 -1 rbd::mirror::NamespaceReplayer: 0x5583f02b2000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.774+0000 7f322d5da700 -1 rbd::mirror::LeaderWatcher: 0x5583ee904c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T23:03:42.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.774+0000 7f322d5da700 -1 rbd::mirror::InstanceWatcher: 0x5583ec3eb180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6194 object: (2) No such file or directory 2024-04-24T23:03:42.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.774+0000 7f32275ce700 -1 rbd::mirror::InstanceWatcher: 0x5583ec3eb180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.774+0000 7f32205c0700 -1 rbd::mirror::MirrorStatusUpdater 0x5583eea24240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.774+0000 7f322d5da700 -1 rbd::mirror::NamespaceReplayer: 0x5583edd62680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.775+0000 7f3226dcd700 -1 rbd::mirror::MirrorStatusUpdater 0x5583eddcf440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.775+0000 7f322dddb700 -1 rbd::mirror::NamespaceReplayer: 0x5583edd62680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.782 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T23:03:42.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:03:42.787 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2' 2024-04-24T23:03:42.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2024-04-24T23:03:42.787 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-24T23:03:42.802 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2024-04-24T23:03:42.802 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:03:42.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.826+0000 7f5ba1f20700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 84594) UID: 1000 2024-04-24T23:03:42.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.826+0000 7f5ba171f700 -1 rbd::mirror::LeaderWatcher: 0x5617a1b3bb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T23:03:42.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.827+0000 7f5ba0f1e700 -1 rbd::mirror::InstanceWatcher: 0x56179e97d500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6772 object: (2) No such file or directory 2024-04-24T23:03:42.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.827+0000 7f5ba0f1e700 -1 rbd::mirror::InstanceWatcher: 0x56179e97d880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6772 object: (2) No such file or directory 2024-04-24T23:03:42.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.827+0000 7f5ba171f700 -1 rbd::mirror::InstanceWatcher: 0x56179e97dc00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6775 object: (2) No such file or directory 2024-04-24T23:03:42.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.827+0000 7f5b9af12700 -1 rbd::mirror::InstanceWatcher: 0x56179e97d500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.827+0000 7f5b9af12700 -1 rbd::mirror::InstanceWatcher: 0x56179e97d880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.827+0000 7f5b8d6f7700 -1 rbd::mirror::InstanceWatcher: 0x56179e97dc00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.828+0000 7f5b94705700 -1 rbd::mirror::MirrorStatusUpdater 0x56179e9e5c20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.828+0000 7f5b94705700 -1 rbd::mirror::MirrorStatusUpdater 0x5617a1028360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.828+0000 7f5ba0f1e700 -1 rbd::mirror::NamespaceReplayer: 0x5617a10261a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.828+0000 7f5ba0f1e700 -1 rbd::mirror::NamespaceReplayer: 0x5617a1026000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.828+0000 7f5b86eea700 -1 rbd::mirror::MirrorStatusUpdater 0x5617a1be4a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.828+0000 7f5ba0f1e700 -1 rbd::mirror::NamespaceReplayer: 0x5617a02f8680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.828+0000 7f5b9a711700 -1 rbd::mirror::MirrorStatusUpdater 0x5617a1028120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.828+0000 7f5b8d6f7700 -1 rbd::mirror::MirrorStatusUpdater 0x5617a1bcbb00 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.828+0000 7f5b9a711700 -1 rbd::mirror::MirrorStatusUpdater 0x5617a1028000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.828+0000 7f5ba171f700 -1 rbd::mirror::NamespaceReplayer: 0x5617a10261a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.828+0000 7f5ba171f700 -1 rbd::mirror::NamespaceReplayer: 0x5617a02f8680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.828+0000 7f5ba171f700 -1 rbd::mirror::NamespaceReplayer: 0x5617a1026000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.829+0000 7f5ba0f1e700 -1 rbd::mirror::LeaderWatcher: 0x5617a0e9ec00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T23:03:42.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.830+0000 7f5ba0f1e700 -1 rbd::mirror::InstanceWatcher: 0x56179e97d180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6772 object: (2) No such file or directory 2024-04-24T23:03:42.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.830+0000 7f5b9a711700 -1 rbd::mirror::InstanceWatcher: 0x56179e97d180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.830+0000 7f5b93f04700 -1 rbd::mirror::MirrorStatusUpdater 0x5617a0fbc240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.830+0000 7f5ba0f1e700 -1 rbd::mirror::NamespaceReplayer: 0x5617a0f161a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.830+0000 7f5b9af12700 -1 rbd::mirror::MirrorStatusUpdater 0x5617a0367440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:2024-04-24T23:03:42.830+0000 7f5ba171f700 -1 rbd::mirror::NamespaceReplayer: 0x5617a0f161a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T23:03:42.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:03:42.842 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1' 2024-04-24T23:03:42.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2024-04-24T23:03:42.842 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-24T23:03:42.854 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2024-04-24T23:03:42.855 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:03:42.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.881+0000 7f87055ba700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 82380) UID: 1000 2024-04-24T23:03:42.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.881+0000 7f8704db9700 -1 rbd::mirror::LeaderWatcher: 0x561a5962a900 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T23:03:42.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.881+0000 7f8704db9700 -1 rbd::mirror::InstanceWatcher: 0x561a557db500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6591 object: (2) No such file or directory 2024-04-24T23:03:42.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.882+0000 7f8704db9700 -1 rbd::mirror::InstanceWatcher: 0x561a557db880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6591 object: (2) No such file or directory 2024-04-24T23:03:42.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.882+0000 7f8704db9700 -1 rbd::mirror::InstanceWatcher: 0x561a557dbc00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6593 object: (2) No such file or directory 2024-04-24T23:03:42.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.882+0000 7f86fe5ac700 -1 rbd::mirror::InstanceWatcher: 0x561a557db500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.882+0000 7f86fddab700 -1 rbd::mirror::InstanceWatcher: 0x561a557db880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.885 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.882+0000 7f86f0d91700 -1 rbd::mirror::InstanceWatcher: 0x561a557dbc00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.882+0000 7f86fe5ac700 -1 rbd::mirror::MirrorStatusUpdater 0x561a58936120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.882+0000 7f87045b8700 -1 rbd::mirror::NamespaceReplayer: 0x561a57e509c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.882+0000 7f86f7d9f700 -1 rbd::mirror::MirrorStatusUpdater 0x561a58936360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.882+0000 7f8704db9700 -1 rbd::mirror::NamespaceReplayer: 0x561a57e50820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.882+0000 7f86e9d83700 -1 rbd::mirror::MirrorStatusUpdater 0x561a5956e6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.882+0000 7f86fddab700 -1 rbd::mirror::MirrorStatusUpdater 0x561a58936000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.882+0000 7f87045b8700 -1 rbd::mirror::NamespaceReplayer: 0x561a596169c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.882+0000 7f87045b8700 -1 rbd::mirror::NamespaceReplayer: 0x561a57e50820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.883+0000 7f86f0590700 -1 rbd::mirror::MirrorStatusUpdater 0x561a58a5bb00 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.883+0000 7f8704db9700 -1 rbd::mirror::NamespaceReplayer: 0x561a596169c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.884+0000 7f87045b8700 -1 rbd::mirror::LeaderWatcher: 0x561a57d9af00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T23:03:42.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.885+0000 7f87045b8700 -1 rbd::mirror::InstanceWatcher: 0x561a557db180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6591 object: (2) No such file or directory 2024-04-24T23:03:42.887 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.885+0000 7f86fddab700 -1 rbd::mirror::InstanceWatcher: 0x561a557db180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.885+0000 7f86f759e700 -1 rbd::mirror::MirrorStatusUpdater 0x561a57e54000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.885+0000 7f87045b8700 -1 rbd::mirror::NamespaceReplayer: 0x561a57e50000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.885+0000 7f86fe5ac700 -1 rbd::mirror::MirrorStatusUpdater 0x561a571f1440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:2024-04-24T23:03:42.885+0000 7f8704db9700 -1 rbd::mirror::NamespaceReplayer: 0x561a57e50000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.892 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T23:03:42.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:03:42.897 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0' 2024-04-24T23:03:42.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2024-04-24T23:03:42.897 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-24T23:03:42.909 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2024-04-24T23:03:42.909 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:03:42.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.936+0000 7fd6b82b0700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 83185) UID: 1000 2024-04-24T23:03:42.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.937+0000 7fd6b7aaf700 -1 rbd::mirror::InstanceWatcher: 0x55bf31b5b500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6072 object: (2) No such file or directory 2024-04-24T23:03:42.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.937+0000 7fd6b72ae700 -1 rbd::mirror::InstanceWatcher: 0x55bf35a40000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6072 object: (2) No such file or directory 2024-04-24T23:03:42.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.937+0000 7fd6a3a87700 -1 rbd::mirror::InstanceWatcher: 0x55bf31b5b880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.937+0000 7fd6b0aa1700 -1 rbd::mirror::InstanceWatcher: 0x55bf31b5b500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.937+0000 7fd6b0aa1700 -1 rbd::mirror::InstanceWatcher: 0x55bf35a40000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.937+0000 7fd6aaa95700 -1 rbd::mirror::MirrorStatusUpdater 0x55bf3420c360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.937+0000 7fd6b7aaf700 -1 rbd::mirror::NamespaceReplayer: 0x55bf3420a000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.938+0000 7fd6aa294700 -1 rbd::mirror::MirrorStatusUpdater 0x55bf31bc3c20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.938+0000 7fd6b72ae700 -1 rbd::mirror::NamespaceReplayer: 0x55bf3420a1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.938+0000 7fd6b12a2700 -1 rbd::mirror::MirrorStatusUpdater 0x55bf3420c000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.938+0000 7fd6b7aaf700 -1 rbd::mirror::NamespaceReplayer: 0x55bf3420a000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.938+0000 7fd6b0aa1700 -1 rbd::mirror::MirrorStatusUpdater 0x55bf3420c120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.938+0000 7fd6b72ae700 -1 rbd::mirror::NamespaceReplayer: 0x55bf3420a1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.939+0000 7fd6b72ae700 -1 rbd::mirror::LeaderWatcher: 0x55bf340fef00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T23:03:42.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.940+0000 7fd6b0aa1700 -1 rbd::mirror::InstanceWatcher: 0x55bf31b5b180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:42.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.940+0000 7fd6aa294700 -1 rbd::mirror::MirrorStatusUpdater 0x55bf341c0000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.940+0000 7fd6b7aaf700 -1 rbd::mirror::NamespaceReplayer: 0x55bf340e84e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.940+0000 7fd6b0aa1700 -1 rbd::mirror::MirrorStatusUpdater 0x55bf3352d0e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:42.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:2024-04-24T23:03:42.940+0000 7fd6b72ae700 -1 rbd::mirror::NamespaceReplayer: 0x55bf340e84e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:42.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T23:03:42.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:03:42.952 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3' 2024-04-24T23:03:42.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2024-04-24T23:03:42.952 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-24T23:03:42.964 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2024-04-24T23:03:42.964 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:03:42.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:2024-04-24T23:03:42.991+0000 7f16e2fec700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 83954) UID: 1000 2024-04-24T23:03:42.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T23:03:43.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:03:43.001 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2' 2024-04-24T23:03:43.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2024-04-24T23:03:43.001 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-24T23:03:43.013 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2024-04-24T23:03:43.013 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:03:43.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:43.040+0000 7fec7bf26700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 84706) UID: 1000 2024-04-24T23:03:43.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:43.041+0000 7fec7af24700 -1 rbd::mirror::InstanceWatcher: 0x557b4856b880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6152 object: (2) No such file or directory 2024-04-24T23:03:43.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:43.041+0000 7fec7af24700 -1 rbd::mirror::InstanceWatcher: 0x557b4856b500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6152 object: (2) No such file or directory 2024-04-24T23:03:43.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:43.041+0000 7fec74f18700 -1 rbd::mirror::InstanceWatcher: 0x557b4856b880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:43.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:43.041+0000 7fec74f18700 -1 rbd::mirror::InstanceWatcher: 0x557b4856b500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:43.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:43.041+0000 7fec6e70b700 -1 rbd::mirror::MirrorStatusUpdater 0x557b4ac0e480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:43.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:43.041+0000 7fec6e70b700 -1 rbd::mirror::MirrorStatusUpdater 0x557b4ac0e360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:43.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:43.041+0000 7fec7af24700 -1 rbd::mirror::NamespaceReplayer: 0x557b4abea1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:43.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:43.041+0000 7fec7af24700 -1 rbd::mirror::NamespaceReplayer: 0x557b4abea000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:43.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:43.041+0000 7fec74f18700 -1 rbd::mirror::MirrorStatusUpdater 0x557b4ac0e120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:43.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:43.042+0000 7fec7b725700 -1 rbd::mirror::NamespaceReplayer: 0x557b4abea1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:43.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:43.042+0000 7fec74f18700 -1 rbd::mirror::MirrorStatusUpdater 0x557b4ac0e000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:43.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:43.042+0000 7fec7af24700 -1 rbd::mirror::NamespaceReplayer: 0x557b4abea000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:44.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:44.293+0000 7fec7b725700 -1 rbd::mirror::LeaderWatcher: 0x557b4c37a900 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T23:03:44.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:44.293+0000 7fec676fd700 -1 rbd::mirror::InstanceWatcher: 0x557b4856bc00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:44.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:44.294+0000 7fec60ef0700 -1 rbd::mirror::MirrorStatusUpdater 0x557b4c2c26c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:44.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:44.294+0000 7fec7af24700 -1 rbd::mirror::NamespaceReplayer: 0x557b4abeab60 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:44.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:44.296+0000 7fec7b725700 -1 rbd::mirror::LeaderWatcher: 0x557b4aaf8f00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T23:03:44.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:44.297+0000 7fec7b725700 -1 rbd::mirror::InstanceWatcher: 0x557b4856b180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6152 object: (2) No such file or directory 2024-04-24T23:03:44.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:44.297+0000 7fec74717700 -1 rbd::mirror::InstanceWatcher: 0x557b4856b180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:44.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:44.297+0000 7fec6e70b700 -1 rbd::mirror::MirrorStatusUpdater 0x557b4ab90240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:44.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:44.297+0000 7fec7b725700 -1 rbd::mirror::NamespaceReplayer: 0x557b4aaea4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:44.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:44.298+0000 7fec74717700 -1 rbd::mirror::MirrorStatusUpdater 0x557b49f59440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:44.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:2024-04-24T23:03:44.298+0000 7fec7af24700 -1 rbd::mirror::NamespaceReplayer: 0x557b4aaea4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:44.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T23:03:45.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:45.918+0000 7f57237ed700 -1 rbd::mirror::InstanceWatcher: 0x55db40ac4388 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:45.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:45.929+0000 7f5737815700 -1 rbd::mirror::LeaderWatcher: 0x55db3fbc3b00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-04-24T23:03:45.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:45.929+0000 7f5722fec700 -1 rbd::mirror::PoolWatcher: 0x55db3c9d03c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T23:03:45.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:45.929+0000 7f571cfe0700 -1 rbd::mirror::PoolWatcher: 0x55db3c9d0000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-24T23:03:45.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:45.929+0000 7f5737014700 -1 rbd::mirror::LeaderWatcher: 0x55db3fbc3b00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-04-24T23:03:45.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:45.929+0000 7f5737014700 -1 rbd::mirror::LeaderWatcher: 0x55db3fbc3b00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-04-24T23:03:49.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:03:49.115 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1' 2024-04-24T23:03:49.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2024-04-24T23:03:49.116 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-24T23:03:49.129 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2024-04-24T23:03:49.129 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:03:49.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:49.129+0000 7f5738016700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 77637) UID: 1000 2024-04-24T23:03:49.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:49.129+0000 7f5737815700 -1 rbd::mirror::LeaderWatcher: 0x55db3fbc3b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-04-24T23:03:49.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:49.130+0000 7f5737815700 -1 rbd::mirror::InstanceWatcher: 0x55db3c9fb500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5654 object: (2) No such file or directory 2024-04-24T23:03:49.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:49.130+0000 7f5737815700 -1 rbd::mirror::InstanceWatcher: 0x55db3c9fb880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5654 object: (2) No such file or directory 2024-04-24T23:03:49.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:49.130+0000 7f5737014700 -1 rbd::mirror::InstanceWatcher: 0x55db3c9fbc00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5657 object: (2) No such file or directory 2024-04-24T23:03:49.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:49.130+0000 7f5731008700 -1 rbd::mirror::InstanceWatcher: 0x55db3c9fb500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:49.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:49.130+0000 7f5731008700 -1 rbd::mirror::InstanceWatcher: 0x55db3c9fb880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:49.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:49.130+0000 7f57237ed700 -1 rbd::mirror::InstanceWatcher: 0x55db3c9fbc00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:49.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:49.131+0000 7f572a7fb700 -1 rbd::mirror::MirrorStatusUpdater 0x55db3f09c360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:49.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:49.131+0000 7f572a7fb700 -1 rbd::mirror::MirrorStatusUpdater 0x55db3ca63c20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:49.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:49.131+0000 7f5737815700 -1 rbd::mirror::NamespaceReplayer: 0x55db3f09a000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:49.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:49.131+0000 7f5737815700 -1 rbd::mirror::NamespaceReplayer: 0x55db3f09a1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:49.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:49.131+0000 7f571cfe0700 -1 rbd::mirror::MirrorStatusUpdater 0x55db408185a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:49.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:49.131+0000 7f5737014700 -1 rbd::mirror::NamespaceReplayer: 0x55db40802340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:49.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:49.131+0000 7f5731008700 -1 rbd::mirror::MirrorStatusUpdater 0x55db3f09c120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:49.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:49.131+0000 7f5737815700 -1 rbd::mirror::NamespaceReplayer: 0x55db3f09a1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:49.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:49.131+0000 7f57237ed700 -1 rbd::mirror::MirrorStatusUpdater 0x55db3fc81e60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:49.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:49.131+0000 7f5737014700 -1 rbd::mirror::NamespaceReplayer: 0x55db40802340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-04-24T23:03:49.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:49.134+0000 7f5730807700 -1 rbd::mirror::InstanceWatcher: 0x55db3c9fb180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-04-24T23:03:49.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:49.134+0000 7f572a7fb700 -1 rbd::mirror::MirrorStatusUpdater 0x55db3f030240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-04-24T23:03:49.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:2024-04-24T23:03:49.134+0000 7f5737815700 -1 rbd::mirror::NamespaceReplayer: 0x55db3e370680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-04-24T23:03:49.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi134.stderr:daemon-helper: command failed with exit status 1 2024-04-24T23:03:49.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:03:49.150 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0' 2024-04-24T23:03:49.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2024-04-24T23:03:49.151 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-24T23:03:49.169 INFO:tasks.ceph.ceph_manager.cluster2:Canceling any pending splits or merges... 2024-04-24T23:03:49.170 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-24T23:03:49.515 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T23:03:49.515 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":508,"fsid":"c413118c-6cbc-48c8-982f-3c229c51647f","created":"2024-04-24T22:28:02.992996+0000","modified":"2024-04-24T23:03:42.030355+0000","last_up_change":"2024-04-24T22:28:11.706725+0000","last_in_change":"2024-04-24T22:28:03.926171+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T22:28:11.879120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T22:28:18.236744+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"15dc0c96-fde9-4f51-a952-d5ed00c63156","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":43461},{"type":"v1","addr":"172.21.15.134:6801","nonce":43461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":43461},{"type":"v1","addr":"172.21.15.134:6803","nonce":43461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":43461},{"type":"v1","addr":"172.21.15.134:6807","nonce":43461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":43461},{"type":"v1","addr":"172.21.15.134:6805","nonce":43461}]},"public_addr":"172.21.15.134:6801/43461","cluster_addr":"172.21.15.134:6803/43461","heartbeat_back_addr":"172.21.15.134:6807/43461","heartbeat_front_addr":"172.21.15.134:6805/43461","state":["exists","up"]},{"osd":1,"uuid":"f65bbd82-d35b-47ae-aa29-588569f95d9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":43460},{"type":"v1","addr":"172.21.15.134:6812","nonce":43460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":43460},{"type":"v1","addr":"172.21.15.134:6816","nonce":43460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6821","nonce":43460},{"type":"v1","addr":"172.21.15.134:6823","nonce":43460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":43460},{"type":"v1","addr":"172.21.15.134:6819","nonce":43460}]},"public_addr":"172.21.15.134:6812/43460","cluster_addr":"172.21.15.134:6816/43460","heartbeat_back_addr":"172.21.15.134:6823/43460","heartbeat_front_addr":"172.21.15.134:6819/43460","state":["exists","up"]},{"osd":2,"uuid":"f097213e-5822-4222-b4b8-a5d430a22da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":43462},{"type":"v1","addr":"172.21.15.134:6809","nonce":43462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6811","nonce":43462},{"type":"v1","addr":"172.21.15.134:6813","nonce":43462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":43462},{"type":"v1","addr":"172.21.15.134:6822","nonce":43462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6815","nonce":43462},{"type":"v1","addr":"172.21.15.134:6817","nonce":43462}]},"public_addr":"172.21.15.134:6809/43462","cluster_addr":"172.21.15.134:6813/43462","heartbeat_back_addr":"172.21.15.134:6822/43462","heartbeat_front_addr":"172.21.15.134:6817/43462","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:28:08.917946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:28:08.976698+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:28:09.060104+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:0/3767159918":"2024-04-25T00:02:07.054558+0000","172.21.15.134:0/1091435556":"2024-04-25T00:00:41.351579+0000","172.21.15.134:0/2684511021":"2024-04-25T00:00:01.284294+0000","172.21.15.134:0/955207952":"2024-04-24T23:59:19.836677+0000","172.21.15.134:0/1527944092":"2024-04-24T23:58:40.657848+0000","172.21.15.134:0/1626435468":"2024-04-24T23:57:16.949918+0000","172.21.15.134:0/1691887091":"2024-04-24T23:53:47.519852+0000","172.21.15.134:0/3719199617":"2024-04-24T23:53:33.534934+0000","172.21.15.134:0/848858089":"2024-04-24T23:53:17.519945+0000","172.21.15.134:0/3855799263":"2024-04-24T23:53:10.486505+0000","172.21.15.134:0/4079769333":"2024-04-24T23:53:25.622005+0000","172.21.15.134:0/320287977":"2024-04-24T23:53:03.461316+0000","172.21.15.134:0/2948758639":"2024-04-24T23:52:19.456864+0000","172.21.15.134:0/1217210943":"2024-04-24T23:52:12.526140+0000","172.21.15.134:0/1898229170":"2024-04-24T23:51:51.458072+0000","172.21.15.134:0/3924791719":"2024-04-24T23:51:44.500103+0000","172.21.15.134:0/1162170109":"2024-04-25T00:01:23.540002+0000","172.21.15.134:0/3944087614":"2024-04-24T23:52:40.461632+0000","172.21.15.134:0/2534860673":"2024-04-24T23:51:37.452348+0000","172.21.15.134:0/205561024":"2024-04-24T23:51:30.475758+0000","172.21.15.134:0/852581212":"2024-04-24T23:52:55.568664+0000","172.21.15.134:0/143447149":"2024-04-24T23:51:23.427757+0000","172.21.15.134:0/1341235345":"2024-04-24T23:52:33.858621+0000","172.21.15.134:0/2036199018":"2024-04-24T23:51:01.478542+0000","172.21.15.134:0/3631255849":"2024-04-24T23:48:31.457857+0000","172.21.15.134:0/2479974152":"2024-04-24T23:51:08.513620+0000","172.21.15.134:0/1504065828":"2024-04-24T23:48:10.443229+0000","172.21.15.134:0/3292703406":"2024-04-24T23:48:03.611226+0000","172.21.15.134:0/1008743850":"2024-04-24T23:49:36.637680+0000","172.21.15.134:0/562444725":"2024-04-24T23:49:00.421865+0000","172.21.15.134:0/3801239299":"2024-04-24T23:38:14.044870+0000","172.21.15.134:0/1408950731":"2024-04-25T00:03:32.599519+0000","172.21.15.134:0/1364571070":"2024-04-24T23:30:34.837052+0000","172.21.15.134:0/3820486962":"2024-04-24T23:47:56.508992+0000","172.21.15.134:0/1531790029":"2024-04-25T00:02:49.550294+0000","172.21.15.134:0/307673385":"2024-04-24T23:37:17.904206+0000","172.21.15.134:0/2651365583":"2024-04-24T23:52:25.712199+0000","172.21.15.134:0/3900431394":"2024-04-24T23:50:47.445637+0000","172.21.15.134:0/1254875856":"2024-04-24T23:50:40.467858+0000","172.21.15.134:0/1087870148":"2024-04-24T23:35:36.132916+0000","172.21.15.134:0/265000914":"2024-04-24T23:50:18.439337+0000","172.21.15.134:0/3244336406":"2024-04-24T23:31:27.950229+0000","172.21.15.134:0/2818517329":"2024-04-24T23:50:04.422106+0000","172.21.15.134:0/4058613801":"2024-04-24T23:52:05.668922+0000","172.21.15.134:0/688534506":"2024-04-24T23:32:16.325841+0000","172.21.15.134:0/2222529796":"2024-04-24T23:34:48.493174+0000","172.21.15.134:0/1790323123":"2024-04-24T23:48:38.526496+0000","172.21.15.134:0/2873122163":"2024-04-24T23:33:07.984615+0000","172.21.15.134:0/178830578":"2024-04-24T23:51:15.518062+0000","172.21.15.134:0/737642934":"2024-04-24T23:48:17.391039+0000","172.21.15.134:0/3349374801":"2024-04-24T23:34:00.122461+0000","172.21.15.134:0/3452451590":"2024-04-24T23:51:57.749526+0000","172.21.15.134:0/374910688":"2024-04-24T23:49:21.428358+0000","172.21.15.134:0/2033608403":"2024-04-24T23:48:45.481841+0000","172.21.15.134:0/733209922":"2024-04-24T23:49:07.467652+0000","172.21.15.134:0/1329902222":"2024-04-24T23:49:14.489367+0000","172.21.15.134:0/381151038":"2024-04-24T23:57:54.485952+0000","172.21.15.134:0/3471946352":"2024-04-24T23:50:11.435248+0000","172.21.15.134:0/388666197":"2024-04-24T23:53:40.579276+0000","172.21.15.134:0/4036221836":"2024-04-24T23:49:43.479513+0000","172.21.15.134:0/2127931850":"2024-04-24T23:49:50.426546+0000","172.21.15.134:0/2503900629":"2024-04-24T23:48:24.437671+0000","172.21.15.134:0/214012243":"2024-04-24T23:49:28.688471+0000","172.21.15.134:0/2372890090":"2024-04-24T23:49:57.432050+0000","172.21.15.134:0/3430268477":"2024-04-24T23:52:47.477404+0000","172.21.15.134:0/4286406172":"2024-04-24T23:36:25.475696+0000","172.21.15.134:0/1948397484":"2024-04-24T23:50:25.508948+0000","172.21.15.134:0/1754247395":"2024-04-24T23:48:53.337888+0000","172.21.15.134:0/1864790492":"2024-04-24T23:50:54.480950+0000","172.21.15.134:0/1300543673":"2024-04-24T23:50:33.419166+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":386,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:03:49.528 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-04-24T23:03:49.528 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-24T23:03:49.836 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T23:03:49.838 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-04-24T23:03:49.849 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":1573,"stamp":"2024-04-24T23:03:48.389943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2977320,"kb_used_data":17788,"kb_used_omap":4,"kb_used_meta":2959483,"kb_avail":280138200,"statfs":{"total":289910292480,"available":286861516800,"internally_reserved":0,"allocated":18214912,"data_stored":10188689,"data_compressed":11512,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":3030510600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2089,"num_read_kb":-1968,"num_write":-409,"num_write_kb":-306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":-4,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.471366"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":500,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.858444+0000","last_change":"2024-04-24T22:28:19.885044+0000","last_active":"2024-04-24T23:03:40.858444+0000","last_peered":"2024-04-24T23:03:40.858444+0000","last_clean":"2024-04-24T23:03:40.858444+0000","last_became_active":"2024-04-24T22:28:19.884405+0000","last_became_peered":"2024-04-24T22:28:19.884405+0000","last_unstale":"2024-04-24T23:03:40.858444+0000","last_undegraded":"2024-04-24T23:03:40.858444+0000","last_fullsized":"2024-04-24T23:03:40.858444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:59:08.229402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":502,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.929185+0000","last_change":"2024-04-24T22:28:19.885719+0000","last_active":"2024-04-24T23:03:39.929185+0000","last_peered":"2024-04-24T23:03:39.929185+0000","last_clean":"2024-04-24T23:03:39.929185+0000","last_became_active":"2024-04-24T22:28:19.885439+0000","last_became_peered":"2024-04-24T22:28:19.885439+0000","last_unstale":"2024-04-24T23:03:39.929185+0000","last_undegraded":"2024-04-24T23:03:39.929185+0000","last_fullsized":"2024-04-24T23:03:39.929185+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:05:50.933126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":501,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:41.430372+0000","last_change":"2024-04-24T22:28:19.883851+0000","last_active":"2024-04-24T23:03:41.430372+0000","last_peered":"2024-04-24T23:03:41.430372+0000","last_clean":"2024-04-24T23:03:41.430372+0000","last_became_active":"2024-04-24T22:28:19.883549+0000","last_became_peered":"2024-04-24T22:28:19.883549+0000","last_unstale":"2024-04-24T23:03:41.430372+0000","last_undegraded":"2024-04-24T23:03:41.430372+0000","last_fullsized":"2024-04-24T23:03:41.430372+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:39:31.099684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":500,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.860533+0000","last_change":"2024-04-24T22:28:19.883883+0000","last_active":"2024-04-24T23:03:40.860533+0000","last_peered":"2024-04-24T23:03:40.860533+0000","last_clean":"2024-04-24T23:03:40.860533+0000","last_became_active":"2024-04-24T22:28:19.883590+0000","last_became_peered":"2024-04-24T22:28:19.883590+0000","last_unstale":"2024-04-24T23:03:40.860533+0000","last_undegraded":"2024-04-24T23:03:40.860533+0000","last_fullsized":"2024-04-24T23:03:40.860533+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:24.775003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":630,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.858703+0000","last_change":"2024-04-24T22:28:13.865092+0000","last_active":"2024-04-24T23:03:40.858703+0000","last_peered":"2024-04-24T23:03:40.858703+0000","last_clean":"2024-04-24T23:03:40.858703+0000","last_became_active":"2024-04-24T22:28:13.864081+0000","last_became_peered":"2024-04-24T22:28:13.864081+0000","last_unstale":"2024-04-24T23:03:40.858703+0000","last_undegraded":"2024-04-24T23:03:40.858703+0000","last_fullsized":"2024-04-24T23:03:40.858703+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:12.854421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:12.854421+0000","last_clean_scrub_stamp":"2024-04-24T22:28:12.854421+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:39:56.775553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":501,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:41.429534+0000","last_change":"2024-04-24T22:28:19.885268+0000","last_active":"2024-04-24T23:03:41.429534+0000","last_peered":"2024-04-24T23:03:41.429534+0000","last_clean":"2024-04-24T23:03:41.429534+0000","last_became_active":"2024-04-24T22:28:19.884948+0000","last_became_peered":"2024-04-24T22:28:19.884948+0000","last_unstale":"2024-04-24T23:03:41.429534+0000","last_undegraded":"2024-04-24T23:03:41.429534+0000","last_fullsized":"2024-04-24T23:03:41.429534+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:00.739359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":500,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.858458+0000","last_change":"2024-04-24T22:28:19.886868+0000","last_active":"2024-04-24T23:03:40.858458+0000","last_peered":"2024-04-24T23:03:40.858458+0000","last_clean":"2024-04-24T23:03:40.858458+0000","last_became_active":"2024-04-24T22:28:19.886592+0000","last_became_peered":"2024-04-24T22:28:19.886592+0000","last_unstale":"2024-04-24T23:03:40.858458+0000","last_undegraded":"2024-04-24T23:03:40.858458+0000","last_fullsized":"2024-04-24T23:03:40.858458+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:50.108365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":500,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.858479+0000","last_change":"2024-04-24T22:28:19.884742+0000","last_active":"2024-04-24T23:03:40.858479+0000","last_peered":"2024-04-24T23:03:40.858479+0000","last_clean":"2024-04-24T23:03:40.858479+0000","last_became_active":"2024-04-24T22:28:19.884428+0000","last_became_peered":"2024-04-24T22:28:19.884428+0000","last_unstale":"2024-04-24T23:03:40.858479+0000","last_undegraded":"2024-04-24T23:03:40.858479+0000","last_fullsized":"2024-04-24T23:03:40.858479+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:12:29.302280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":500,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.858493+0000","last_change":"2024-04-24T22:28:19.886666+0000","last_active":"2024-04-24T23:03:40.858493+0000","last_peered":"2024-04-24T23:03:40.858493+0000","last_clean":"2024-04-24T23:03:40.858493+0000","last_became_active":"2024-04-24T22:28:19.886369+0000","last_became_peered":"2024-04-24T22:28:19.886369+0000","last_unstale":"2024-04-24T23:03:40.858493+0000","last_undegraded":"2024-04-24T23:03:40.858493+0000","last_fullsized":"2024-04-24T23:03:40.858493+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:19:47.445279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11512,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706095,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":884052,"kb_used_data":5892,"kb_used_omap":1,"kb_used_meta":878142,"kb_avail":93487788,"statfs":{"total":96636764160,"available":95731494912,"internally_reserved":0,"allocated":6033408,"data_stored":3002651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":899217752},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 23:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.82099999999999995,"15min":0.504},"min":{"1min":0.154,"5min":0.154,"15min":0.14699999999999999},"max":{"1min":5.7530000000000001,"5min":12.317,"15min":12.317},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.73099999999999998,"15min":0.46899999999999997},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.16900000000000001},"max":{"1min":3.9910000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"last":0.33300000000000002}]},{"osd":1,"last update":"Wed Apr 24 23:03:02 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":0.71599999999999997,"15min":0.48899999999999999},"min":{"1min":0.25700000000000001,"5min":0.185,"15min":0.18099999999999999},"max":{"1min":8.9510000000000005,"5min":8.9510000000000005,"15min":8.9510000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.68899999999999995,"15min":0.46500000000000002},"min":{"1min":0.254,"5min":0.20200000000000001,"15min":0.189},"max":{"1min":3.4689999999999999,"5min":5.4249999999999998,"15min":5.4249999999999998},"last":0.34899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738797,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1200968,"kb_used_data":5948,"kb_used_omap":1,"kb_used_meta":1195006,"kb_avail":93170872,"statfs":{"total":96636764160,"available":95406972928,"internally_reserved":0,"allocated":6090752,"data_stored":3593019,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":1223686488},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 23:03:31 2024","interfaces":[{"interface":"back","average":{"1min":1.752,"5min":0.99399999999999999,"15min":0.56599999999999995},"min":{"1min":0.247,"5min":0.156,"15min":0.156},"max":{"1min":13.428000000000001,"5min":13.428000000000001,"15min":13.428000000000001},"last":0.442},{"interface":"front","average":{"1min":1.0980000000000001,"5min":0.83299999999999996,"15min":0.53300000000000003},"min":{"1min":0.20599999999999999,"5min":0.19500000000000001,"15min":0.18099999999999999},"max":{"1min":4.6299999999999999,"5min":7.5279999999999996,"15min":7.5279999999999996},"last":0.51200000000000001}]},{"osd":2,"last update":"Wed Apr 24 23:03:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":1.0329999999999999,"15min":0.58299999999999996},"min":{"1min":0.249,"5min":0.184,"15min":0.184},"max":{"1min":13.478,"5min":13.478,"15min":13.478},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.60299999999999998,"15min":0.45300000000000001},"min":{"1min":0.27000000000000002,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":2,"5min":6.7130000000000001,"15min":6.7130000000000001},"last":0.47799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738797,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":892300,"kb_used_data":5948,"kb_used_omap":1,"kb_used_meta":886334,"kb_avail":93479540,"statfs":{"total":96636764160,"available":95723048960,"internally_reserved":0,"allocated":6090752,"data_stored":3593019,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":907606360},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 23:03:29 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":0.84999999999999998,"15min":0.55800000000000005},"min":{"1min":0.23699999999999999,"5min":0.155,"15min":0.155},"max":{"1min":14.769,"5min":14.769,"15min":14.769},"last":0.374},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61799999999999999,"15min":0.45000000000000001},"min":{"1min":0.26400000000000001,"5min":0.186,"15min":0.186},"max":{"1min":3.8050000000000002,"5min":7.4539999999999997,"15min":7.4539999999999997},"last":0.35699999999999998}]},{"osd":2,"last update":"Wed Apr 24 23:03:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.65500000000000003,"15min":0.53400000000000003},"min":{"1min":0.25800000000000001,"5min":0.182,"15min":0.182},"max":{"1min":4.5670000000000002,"5min":6.4299999999999997,"15min":15.769},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.66700000000000004,"15min":0.44500000000000001},"min":{"1min":0.252,"5min":0.20300000000000001,"15min":0.185},"max":{"1min":2.0619999999999998,"5min":8.3550000000000004,"15min":8.3550000000000004},"last":0.40200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:03:49.850 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-24T23:03:50.158 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T23:03:50.160 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-04-24T23:03:50.171 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":1573,"stamp":"2024-04-24T23:03:48.389943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2977320,"kb_used_data":17788,"kb_used_omap":4,"kb_used_meta":2959483,"kb_avail":280138200,"statfs":{"total":289910292480,"available":286861516800,"internally_reserved":0,"allocated":18214912,"data_stored":10188689,"data_compressed":11512,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":3030510600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2089,"num_read_kb":-1968,"num_write":-409,"num_write_kb":-306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":-4,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.471366"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":500,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.858444+0000","last_change":"2024-04-24T22:28:19.885044+0000","last_active":"2024-04-24T23:03:40.858444+0000","last_peered":"2024-04-24T23:03:40.858444+0000","last_clean":"2024-04-24T23:03:40.858444+0000","last_became_active":"2024-04-24T22:28:19.884405+0000","last_became_peered":"2024-04-24T22:28:19.884405+0000","last_unstale":"2024-04-24T23:03:40.858444+0000","last_undegraded":"2024-04-24T23:03:40.858444+0000","last_fullsized":"2024-04-24T23:03:40.858444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:59:08.229402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":502,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.929185+0000","last_change":"2024-04-24T22:28:19.885719+0000","last_active":"2024-04-24T23:03:39.929185+0000","last_peered":"2024-04-24T23:03:39.929185+0000","last_clean":"2024-04-24T23:03:39.929185+0000","last_became_active":"2024-04-24T22:28:19.885439+0000","last_became_peered":"2024-04-24T22:28:19.885439+0000","last_unstale":"2024-04-24T23:03:39.929185+0000","last_undegraded":"2024-04-24T23:03:39.929185+0000","last_fullsized":"2024-04-24T23:03:39.929185+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:05:50.933126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":501,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:41.430372+0000","last_change":"2024-04-24T22:28:19.883851+0000","last_active":"2024-04-24T23:03:41.430372+0000","last_peered":"2024-04-24T23:03:41.430372+0000","last_clean":"2024-04-24T23:03:41.430372+0000","last_became_active":"2024-04-24T22:28:19.883549+0000","last_became_peered":"2024-04-24T22:28:19.883549+0000","last_unstale":"2024-04-24T23:03:41.430372+0000","last_undegraded":"2024-04-24T23:03:41.430372+0000","last_fullsized":"2024-04-24T23:03:41.430372+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:39:31.099684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":500,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.860533+0000","last_change":"2024-04-24T22:28:19.883883+0000","last_active":"2024-04-24T23:03:40.860533+0000","last_peered":"2024-04-24T23:03:40.860533+0000","last_clean":"2024-04-24T23:03:40.860533+0000","last_became_active":"2024-04-24T22:28:19.883590+0000","last_became_peered":"2024-04-24T22:28:19.883590+0000","last_unstale":"2024-04-24T23:03:40.860533+0000","last_undegraded":"2024-04-24T23:03:40.860533+0000","last_fullsized":"2024-04-24T23:03:40.860533+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:24.775003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":630,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.858703+0000","last_change":"2024-04-24T22:28:13.865092+0000","last_active":"2024-04-24T23:03:40.858703+0000","last_peered":"2024-04-24T23:03:40.858703+0000","last_clean":"2024-04-24T23:03:40.858703+0000","last_became_active":"2024-04-24T22:28:13.864081+0000","last_became_peered":"2024-04-24T22:28:13.864081+0000","last_unstale":"2024-04-24T23:03:40.858703+0000","last_undegraded":"2024-04-24T23:03:40.858703+0000","last_fullsized":"2024-04-24T23:03:40.858703+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:12.854421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:12.854421+0000","last_clean_scrub_stamp":"2024-04-24T22:28:12.854421+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:39:56.775553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":501,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:41.429534+0000","last_change":"2024-04-24T22:28:19.885268+0000","last_active":"2024-04-24T23:03:41.429534+0000","last_peered":"2024-04-24T23:03:41.429534+0000","last_clean":"2024-04-24T23:03:41.429534+0000","last_became_active":"2024-04-24T22:28:19.884948+0000","last_became_peered":"2024-04-24T22:28:19.884948+0000","last_unstale":"2024-04-24T23:03:41.429534+0000","last_undegraded":"2024-04-24T23:03:41.429534+0000","last_fullsized":"2024-04-24T23:03:41.429534+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:00.739359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":500,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.858458+0000","last_change":"2024-04-24T22:28:19.886868+0000","last_active":"2024-04-24T23:03:40.858458+0000","last_peered":"2024-04-24T23:03:40.858458+0000","last_clean":"2024-04-24T23:03:40.858458+0000","last_became_active":"2024-04-24T22:28:19.886592+0000","last_became_peered":"2024-04-24T22:28:19.886592+0000","last_unstale":"2024-04-24T23:03:40.858458+0000","last_undegraded":"2024-04-24T23:03:40.858458+0000","last_fullsized":"2024-04-24T23:03:40.858458+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:50.108365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":500,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.858479+0000","last_change":"2024-04-24T22:28:19.884742+0000","last_active":"2024-04-24T23:03:40.858479+0000","last_peered":"2024-04-24T23:03:40.858479+0000","last_clean":"2024-04-24T23:03:40.858479+0000","last_became_active":"2024-04-24T22:28:19.884428+0000","last_became_peered":"2024-04-24T22:28:19.884428+0000","last_unstale":"2024-04-24T23:03:40.858479+0000","last_undegraded":"2024-04-24T23:03:40.858479+0000","last_fullsized":"2024-04-24T23:03:40.858479+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:12:29.302280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":500,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.858493+0000","last_change":"2024-04-24T22:28:19.886666+0000","last_active":"2024-04-24T23:03:40.858493+0000","last_peered":"2024-04-24T23:03:40.858493+0000","last_clean":"2024-04-24T23:03:40.858493+0000","last_became_active":"2024-04-24T22:28:19.886369+0000","last_became_peered":"2024-04-24T22:28:19.886369+0000","last_unstale":"2024-04-24T23:03:40.858493+0000","last_undegraded":"2024-04-24T23:03:40.858493+0000","last_fullsized":"2024-04-24T23:03:40.858493+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:19:47.445279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11512,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706095,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":884052,"kb_used_data":5892,"kb_used_omap":1,"kb_used_meta":878142,"kb_avail":93487788,"statfs":{"total":96636764160,"available":95731494912,"internally_reserved":0,"allocated":6033408,"data_stored":3002651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":899217752},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 23:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.82099999999999995,"15min":0.504},"min":{"1min":0.154,"5min":0.154,"15min":0.14699999999999999},"max":{"1min":5.7530000000000001,"5min":12.317,"15min":12.317},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.73099999999999998,"15min":0.46899999999999997},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.16900000000000001},"max":{"1min":3.9910000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"last":0.33300000000000002}]},{"osd":1,"last update":"Wed Apr 24 23:03:02 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":0.71599999999999997,"15min":0.48899999999999999},"min":{"1min":0.25700000000000001,"5min":0.185,"15min":0.18099999999999999},"max":{"1min":8.9510000000000005,"5min":8.9510000000000005,"15min":8.9510000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.68899999999999995,"15min":0.46500000000000002},"min":{"1min":0.254,"5min":0.20200000000000001,"15min":0.189},"max":{"1min":3.4689999999999999,"5min":5.4249999999999998,"15min":5.4249999999999998},"last":0.34899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738797,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1200968,"kb_used_data":5948,"kb_used_omap":1,"kb_used_meta":1195006,"kb_avail":93170872,"statfs":{"total":96636764160,"available":95406972928,"internally_reserved":0,"allocated":6090752,"data_stored":3593019,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":1223686488},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 23:03:31 2024","interfaces":[{"interface":"back","average":{"1min":1.752,"5min":0.99399999999999999,"15min":0.56599999999999995},"min":{"1min":0.247,"5min":0.156,"15min":0.156},"max":{"1min":13.428000000000001,"5min":13.428000000000001,"15min":13.428000000000001},"last":0.442},{"interface":"front","average":{"1min":1.0980000000000001,"5min":0.83299999999999996,"15min":0.53300000000000003},"min":{"1min":0.20599999999999999,"5min":0.19500000000000001,"15min":0.18099999999999999},"max":{"1min":4.6299999999999999,"5min":7.5279999999999996,"15min":7.5279999999999996},"last":0.51200000000000001}]},{"osd":2,"last update":"Wed Apr 24 23:03:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":1.0329999999999999,"15min":0.58299999999999996},"min":{"1min":0.249,"5min":0.184,"15min":0.184},"max":{"1min":13.478,"5min":13.478,"15min":13.478},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.60299999999999998,"15min":0.45300000000000001},"min":{"1min":0.27000000000000002,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":2,"5min":6.7130000000000001,"15min":6.7130000000000001},"last":0.47799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738797,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":892300,"kb_used_data":5948,"kb_used_omap":1,"kb_used_meta":886334,"kb_avail":93479540,"statfs":{"total":96636764160,"available":95723048960,"internally_reserved":0,"allocated":6090752,"data_stored":3593019,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":907606360},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 23:03:29 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":0.84999999999999998,"15min":0.55800000000000005},"min":{"1min":0.23699999999999999,"5min":0.155,"15min":0.155},"max":{"1min":14.769,"5min":14.769,"15min":14.769},"last":0.374},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61799999999999999,"15min":0.45000000000000001},"min":{"1min":0.26400000000000001,"5min":0.186,"15min":0.186},"max":{"1min":3.8050000000000002,"5min":7.4539999999999997,"15min":7.4539999999999997},"last":0.35699999999999998}]},{"osd":2,"last update":"Wed Apr 24 23:03:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.65500000000000003,"15min":0.53400000000000003},"min":{"1min":0.25800000000000001,"5min":0.182,"15min":0.182},"max":{"1min":4.5670000000000002,"5min":6.4299999999999997,"15min":15.769},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.66700000000000004,"15min":0.44500000000000001},"min":{"1min":0.252,"5min":0.20300000000000001,"15min":0.185},"max":{"1min":2.0619999999999998,"5min":8.3550000000000004,"15min":8.3550000000000004},"last":0.40200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:03:50.172 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-04-24T23:03:50.172 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-24T23:03:50.477 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T23:03:50.480 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-04-24T23:03:50.491 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":1574,"stamp":"2024-04-24T23:03:50.390280+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2977320,"kb_used_data":17788,"kb_used_omap":4,"kb_used_meta":2959483,"kb_avail":280138200,"statfs":{"total":289910292480,"available":286861516800,"internally_reserved":0,"allocated":18214912,"data_stored":10188689,"data_compressed":11512,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":3030510600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2089,"num_read_kb":-1968,"num_write":-409,"num_write_kb":-306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":-4,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001851"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":500,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.858444+0000","last_change":"2024-04-24T22:28:19.885044+0000","last_active":"2024-04-24T23:03:40.858444+0000","last_peered":"2024-04-24T23:03:40.858444+0000","last_clean":"2024-04-24T23:03:40.858444+0000","last_became_active":"2024-04-24T22:28:19.884405+0000","last_became_peered":"2024-04-24T22:28:19.884405+0000","last_unstale":"2024-04-24T23:03:40.858444+0000","last_undegraded":"2024-04-24T23:03:40.858444+0000","last_fullsized":"2024-04-24T23:03:40.858444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:59:08.229402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":502,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.929185+0000","last_change":"2024-04-24T22:28:19.885719+0000","last_active":"2024-04-24T23:03:39.929185+0000","last_peered":"2024-04-24T23:03:39.929185+0000","last_clean":"2024-04-24T23:03:39.929185+0000","last_became_active":"2024-04-24T22:28:19.885439+0000","last_became_peered":"2024-04-24T22:28:19.885439+0000","last_unstale":"2024-04-24T23:03:39.929185+0000","last_undegraded":"2024-04-24T23:03:39.929185+0000","last_fullsized":"2024-04-24T23:03:39.929185+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:05:50.933126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":501,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:41.430372+0000","last_change":"2024-04-24T22:28:19.883851+0000","last_active":"2024-04-24T23:03:41.430372+0000","last_peered":"2024-04-24T23:03:41.430372+0000","last_clean":"2024-04-24T23:03:41.430372+0000","last_became_active":"2024-04-24T22:28:19.883549+0000","last_became_peered":"2024-04-24T22:28:19.883549+0000","last_unstale":"2024-04-24T23:03:41.430372+0000","last_undegraded":"2024-04-24T23:03:41.430372+0000","last_fullsized":"2024-04-24T23:03:41.430372+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:39:31.099684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":500,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.860533+0000","last_change":"2024-04-24T22:28:19.883883+0000","last_active":"2024-04-24T23:03:40.860533+0000","last_peered":"2024-04-24T23:03:40.860533+0000","last_clean":"2024-04-24T23:03:40.860533+0000","last_became_active":"2024-04-24T22:28:19.883590+0000","last_became_peered":"2024-04-24T22:28:19.883590+0000","last_unstale":"2024-04-24T23:03:40.860533+0000","last_undegraded":"2024-04-24T23:03:40.860533+0000","last_fullsized":"2024-04-24T23:03:40.860533+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:24.775003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":630,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.858703+0000","last_change":"2024-04-24T22:28:13.865092+0000","last_active":"2024-04-24T23:03:40.858703+0000","last_peered":"2024-04-24T23:03:40.858703+0000","last_clean":"2024-04-24T23:03:40.858703+0000","last_became_active":"2024-04-24T22:28:13.864081+0000","last_became_peered":"2024-04-24T22:28:13.864081+0000","last_unstale":"2024-04-24T23:03:40.858703+0000","last_undegraded":"2024-04-24T23:03:40.858703+0000","last_fullsized":"2024-04-24T23:03:40.858703+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:12.854421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:12.854421+0000","last_clean_scrub_stamp":"2024-04-24T22:28:12.854421+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:39:56.775553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":501,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:41.429534+0000","last_change":"2024-04-24T22:28:19.885268+0000","last_active":"2024-04-24T23:03:41.429534+0000","last_peered":"2024-04-24T23:03:41.429534+0000","last_clean":"2024-04-24T23:03:41.429534+0000","last_became_active":"2024-04-24T22:28:19.884948+0000","last_became_peered":"2024-04-24T22:28:19.884948+0000","last_unstale":"2024-04-24T23:03:41.429534+0000","last_undegraded":"2024-04-24T23:03:41.429534+0000","last_fullsized":"2024-04-24T23:03:41.429534+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:00.739359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":500,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.858458+0000","last_change":"2024-04-24T22:28:19.886868+0000","last_active":"2024-04-24T23:03:40.858458+0000","last_peered":"2024-04-24T23:03:40.858458+0000","last_clean":"2024-04-24T23:03:40.858458+0000","last_became_active":"2024-04-24T22:28:19.886592+0000","last_became_peered":"2024-04-24T22:28:19.886592+0000","last_unstale":"2024-04-24T23:03:40.858458+0000","last_undegraded":"2024-04-24T23:03:40.858458+0000","last_fullsized":"2024-04-24T23:03:40.858458+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:50.108365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":500,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.858479+0000","last_change":"2024-04-24T22:28:19.884742+0000","last_active":"2024-04-24T23:03:40.858479+0000","last_peered":"2024-04-24T23:03:40.858479+0000","last_clean":"2024-04-24T23:03:40.858479+0000","last_became_active":"2024-04-24T22:28:19.884428+0000","last_became_peered":"2024-04-24T22:28:19.884428+0000","last_unstale":"2024-04-24T23:03:40.858479+0000","last_undegraded":"2024-04-24T23:03:40.858479+0000","last_fullsized":"2024-04-24T23:03:40.858479+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:12:29.302280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":500,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.858493+0000","last_change":"2024-04-24T22:28:19.886666+0000","last_active":"2024-04-24T23:03:40.858493+0000","last_peered":"2024-04-24T23:03:40.858493+0000","last_clean":"2024-04-24T23:03:40.858493+0000","last_became_active":"2024-04-24T22:28:19.886369+0000","last_became_peered":"2024-04-24T22:28:19.886369+0000","last_unstale":"2024-04-24T23:03:40.858493+0000","last_undegraded":"2024-04-24T23:03:40.858493+0000","last_fullsized":"2024-04-24T23:03:40.858493+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:19:47.445279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11512,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706095,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":884052,"kb_used_data":5892,"kb_used_omap":1,"kb_used_meta":878142,"kb_avail":93487788,"statfs":{"total":96636764160,"available":95731494912,"internally_reserved":0,"allocated":6033408,"data_stored":3002651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":899217752},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 23:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.82099999999999995,"15min":0.504},"min":{"1min":0.154,"5min":0.154,"15min":0.14699999999999999},"max":{"1min":5.7530000000000001,"5min":12.317,"15min":12.317},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.73099999999999998,"15min":0.46899999999999997},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.16900000000000001},"max":{"1min":3.9910000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"last":0.33300000000000002}]},{"osd":1,"last update":"Wed Apr 24 23:03:02 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":0.71599999999999997,"15min":0.48899999999999999},"min":{"1min":0.25700000000000001,"5min":0.185,"15min":0.18099999999999999},"max":{"1min":8.9510000000000005,"5min":8.9510000000000005,"15min":8.9510000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.68899999999999995,"15min":0.46500000000000002},"min":{"1min":0.254,"5min":0.20200000000000001,"15min":0.189},"max":{"1min":3.4689999999999999,"5min":5.4249999999999998,"15min":5.4249999999999998},"last":0.34899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738797,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1200968,"kb_used_data":5948,"kb_used_omap":1,"kb_used_meta":1195006,"kb_avail":93170872,"statfs":{"total":96636764160,"available":95406972928,"internally_reserved":0,"allocated":6090752,"data_stored":3593019,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":1223686488},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 23:03:31 2024","interfaces":[{"interface":"back","average":{"1min":1.752,"5min":0.99399999999999999,"15min":0.56599999999999995},"min":{"1min":0.247,"5min":0.156,"15min":0.156},"max":{"1min":13.428000000000001,"5min":13.428000000000001,"15min":13.428000000000001},"last":0.442},{"interface":"front","average":{"1min":1.0980000000000001,"5min":0.83299999999999996,"15min":0.53300000000000003},"min":{"1min":0.20599999999999999,"5min":0.19500000000000001,"15min":0.18099999999999999},"max":{"1min":4.6299999999999999,"5min":7.5279999999999996,"15min":7.5279999999999996},"last":0.51200000000000001}]},{"osd":2,"last update":"Wed Apr 24 23:03:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":1.0329999999999999,"15min":0.58299999999999996},"min":{"1min":0.249,"5min":0.184,"15min":0.184},"max":{"1min":13.478,"5min":13.478,"15min":13.478},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.60299999999999998,"15min":0.45300000000000001},"min":{"1min":0.27000000000000002,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":2,"5min":6.7130000000000001,"15min":6.7130000000000001},"last":0.47799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738797,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":892300,"kb_used_data":5948,"kb_used_omap":1,"kb_used_meta":886334,"kb_avail":93479540,"statfs":{"total":96636764160,"available":95723048960,"internally_reserved":0,"allocated":6090752,"data_stored":3593019,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":907606360},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 23:03:29 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":0.84999999999999998,"15min":0.55800000000000005},"min":{"1min":0.23699999999999999,"5min":0.155,"15min":0.155},"max":{"1min":14.769,"5min":14.769,"15min":14.769},"last":0.374},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61799999999999999,"15min":0.45000000000000001},"min":{"1min":0.26400000000000001,"5min":0.186,"15min":0.186},"max":{"1min":3.8050000000000002,"5min":7.4539999999999997,"15min":7.4539999999999997},"last":0.35699999999999998}]},{"osd":2,"last update":"Wed Apr 24 23:03:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.65500000000000003,"15min":0.53400000000000003},"min":{"1min":0.25800000000000001,"5min":0.182,"15min":0.182},"max":{"1min":4.5670000000000002,"5min":6.4299999999999997,"15min":15.769},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.66700000000000004,"15min":0.44500000000000001},"min":{"1min":0.252,"5min":0.20300000000000001,"15min":0.185},"max":{"1min":2.0619999999999998,"5min":8.3550000000000004,"15min":8.3550000000000004},"last":0.40200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:03:50.492 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-24T23:03:50.805 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T23:03:50.805 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":508,"fsid":"c413118c-6cbc-48c8-982f-3c229c51647f","created":"2024-04-24T22:28:02.992996+0000","modified":"2024-04-24T23:03:42.030355+0000","last_up_change":"2024-04-24T22:28:11.706725+0000","last_in_change":"2024-04-24T22:28:03.926171+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T22:28:11.879120+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T22:28:18.236744+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"15dc0c96-fde9-4f51-a952-d5ed00c63156","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":43461},{"type":"v1","addr":"172.21.15.134:6801","nonce":43461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":43461},{"type":"v1","addr":"172.21.15.134:6803","nonce":43461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":43461},{"type":"v1","addr":"172.21.15.134:6807","nonce":43461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":43461},{"type":"v1","addr":"172.21.15.134:6805","nonce":43461}]},"public_addr":"172.21.15.134:6801/43461","cluster_addr":"172.21.15.134:6803/43461","heartbeat_back_addr":"172.21.15.134:6807/43461","heartbeat_front_addr":"172.21.15.134:6805/43461","state":["exists","up"]},{"osd":1,"uuid":"f65bbd82-d35b-47ae-aa29-588569f95d9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":43460},{"type":"v1","addr":"172.21.15.134:6812","nonce":43460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":43460},{"type":"v1","addr":"172.21.15.134:6816","nonce":43460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6821","nonce":43460},{"type":"v1","addr":"172.21.15.134:6823","nonce":43460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":43460},{"type":"v1","addr":"172.21.15.134:6819","nonce":43460}]},"public_addr":"172.21.15.134:6812/43460","cluster_addr":"172.21.15.134:6816/43460","heartbeat_back_addr":"172.21.15.134:6823/43460","heartbeat_front_addr":"172.21.15.134:6819/43460","state":["exists","up"]},{"osd":2,"uuid":"f097213e-5822-4222-b4b8-a5d430a22da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":43462},{"type":"v1","addr":"172.21.15.134:6809","nonce":43462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6811","nonce":43462},{"type":"v1","addr":"172.21.15.134:6813","nonce":43462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":43462},{"type":"v1","addr":"172.21.15.134:6822","nonce":43462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6815","nonce":43462},{"type":"v1","addr":"172.21.15.134:6817","nonce":43462}]},"public_addr":"172.21.15.134:6809/43462","cluster_addr":"172.21.15.134:6813/43462","heartbeat_back_addr":"172.21.15.134:6822/43462","heartbeat_front_addr":"172.21.15.134:6817/43462","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:28:08.917946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:28:08.976698+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:28:09.060104+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:0/3767159918":"2024-04-25T00:02:07.054558+0000","172.21.15.134:0/1091435556":"2024-04-25T00:00:41.351579+0000","172.21.15.134:0/2684511021":"2024-04-25T00:00:01.284294+0000","172.21.15.134:0/955207952":"2024-04-24T23:59:19.836677+0000","172.21.15.134:0/1527944092":"2024-04-24T23:58:40.657848+0000","172.21.15.134:0/1626435468":"2024-04-24T23:57:16.949918+0000","172.21.15.134:0/1691887091":"2024-04-24T23:53:47.519852+0000","172.21.15.134:0/3719199617":"2024-04-24T23:53:33.534934+0000","172.21.15.134:0/848858089":"2024-04-24T23:53:17.519945+0000","172.21.15.134:0/3855799263":"2024-04-24T23:53:10.486505+0000","172.21.15.134:0/4079769333":"2024-04-24T23:53:25.622005+0000","172.21.15.134:0/320287977":"2024-04-24T23:53:03.461316+0000","172.21.15.134:0/2948758639":"2024-04-24T23:52:19.456864+0000","172.21.15.134:0/1217210943":"2024-04-24T23:52:12.526140+0000","172.21.15.134:0/1898229170":"2024-04-24T23:51:51.458072+0000","172.21.15.134:0/3924791719":"2024-04-24T23:51:44.500103+0000","172.21.15.134:0/1162170109":"2024-04-25T00:01:23.540002+0000","172.21.15.134:0/3944087614":"2024-04-24T23:52:40.461632+0000","172.21.15.134:0/2534860673":"2024-04-24T23:51:37.452348+0000","172.21.15.134:0/205561024":"2024-04-24T23:51:30.475758+0000","172.21.15.134:0/852581212":"2024-04-24T23:52:55.568664+0000","172.21.15.134:0/143447149":"2024-04-24T23:51:23.427757+0000","172.21.15.134:0/1341235345":"2024-04-24T23:52:33.858621+0000","172.21.15.134:0/2036199018":"2024-04-24T23:51:01.478542+0000","172.21.15.134:0/3631255849":"2024-04-24T23:48:31.457857+0000","172.21.15.134:0/2479974152":"2024-04-24T23:51:08.513620+0000","172.21.15.134:0/1504065828":"2024-04-24T23:48:10.443229+0000","172.21.15.134:0/3292703406":"2024-04-24T23:48:03.611226+0000","172.21.15.134:0/1008743850":"2024-04-24T23:49:36.637680+0000","172.21.15.134:0/562444725":"2024-04-24T23:49:00.421865+0000","172.21.15.134:0/3801239299":"2024-04-24T23:38:14.044870+0000","172.21.15.134:0/1408950731":"2024-04-25T00:03:32.599519+0000","172.21.15.134:0/1364571070":"2024-04-24T23:30:34.837052+0000","172.21.15.134:0/3820486962":"2024-04-24T23:47:56.508992+0000","172.21.15.134:0/1531790029":"2024-04-25T00:02:49.550294+0000","172.21.15.134:0/307673385":"2024-04-24T23:37:17.904206+0000","172.21.15.134:0/2651365583":"2024-04-24T23:52:25.712199+0000","172.21.15.134:0/3900431394":"2024-04-24T23:50:47.445637+0000","172.21.15.134:0/1254875856":"2024-04-24T23:50:40.467858+0000","172.21.15.134:0/1087870148":"2024-04-24T23:35:36.132916+0000","172.21.15.134:0/265000914":"2024-04-24T23:50:18.439337+0000","172.21.15.134:0/3244336406":"2024-04-24T23:31:27.950229+0000","172.21.15.134:0/2818517329":"2024-04-24T23:50:04.422106+0000","172.21.15.134:0/4058613801":"2024-04-24T23:52:05.668922+0000","172.21.15.134:0/688534506":"2024-04-24T23:32:16.325841+0000","172.21.15.134:0/2222529796":"2024-04-24T23:34:48.493174+0000","172.21.15.134:0/1790323123":"2024-04-24T23:48:38.526496+0000","172.21.15.134:0/2873122163":"2024-04-24T23:33:07.984615+0000","172.21.15.134:0/178830578":"2024-04-24T23:51:15.518062+0000","172.21.15.134:0/737642934":"2024-04-24T23:48:17.391039+0000","172.21.15.134:0/3349374801":"2024-04-24T23:34:00.122461+0000","172.21.15.134:0/3452451590":"2024-04-24T23:51:57.749526+0000","172.21.15.134:0/374910688":"2024-04-24T23:49:21.428358+0000","172.21.15.134:0/2033608403":"2024-04-24T23:48:45.481841+0000","172.21.15.134:0/733209922":"2024-04-24T23:49:07.467652+0000","172.21.15.134:0/1329902222":"2024-04-24T23:49:14.489367+0000","172.21.15.134:0/381151038":"2024-04-24T23:57:54.485952+0000","172.21.15.134:0/3471946352":"2024-04-24T23:50:11.435248+0000","172.21.15.134:0/388666197":"2024-04-24T23:53:40.579276+0000","172.21.15.134:0/4036221836":"2024-04-24T23:49:43.479513+0000","172.21.15.134:0/2127931850":"2024-04-24T23:49:50.426546+0000","172.21.15.134:0/2503900629":"2024-04-24T23:48:24.437671+0000","172.21.15.134:0/214012243":"2024-04-24T23:49:28.688471+0000","172.21.15.134:0/2372890090":"2024-04-24T23:49:57.432050+0000","172.21.15.134:0/3430268477":"2024-04-24T23:52:47.477404+0000","172.21.15.134:0/4286406172":"2024-04-24T23:36:25.475696+0000","172.21.15.134:0/1948397484":"2024-04-24T23:50:25.508948+0000","172.21.15.134:0/1754247395":"2024-04-24T23:48:53.337888+0000","172.21.15.134:0/1864790492":"2024-04-24T23:50:54.480950+0000","172.21.15.134:0/1300543673":"2024-04-24T23:50:33.419166+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":386,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:03:51.820 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-04-24T23:03:51.820 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-24T23:03:51.974 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-04-24T23:03:51.974 INFO:teuthology.orchestra.run.smithi134.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T23:03:51.974 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-04-24T23:03:51.986 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-04-24T23:03:52.299 INFO:teuthology.orchestra.run.smithi134.stderr:instructed osd(s) 0 to deep-scrub 2024-04-24T23:03:52.310 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-04-24T23:03:52.311 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-24T23:03:52.450 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-04-24T23:03:52.450 INFO:teuthology.orchestra.run.smithi134.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T23:03:52.451 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-04-24T23:03:52.462 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-04-24T23:03:52.781 INFO:teuthology.orchestra.run.smithi134.stderr:instructed osd(s) 1 to deep-scrub 2024-04-24T23:03:52.792 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-04-24T23:03:52.792 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-24T23:03:52.933 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-04-24T23:03:52.933 INFO:teuthology.orchestra.run.smithi134.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T23:03:52.933 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-04-24T23:03:52.944 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-04-24T23:03:53.257 INFO:teuthology.orchestra.run.smithi134.stderr:instructed osd(s) 2 to deep-scrub 2024-04-24T23:03:53.268 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-24T23:03:53.575 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T23:03:53.577 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-04-24T23:03:53.589 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":1575,"stamp":"2024-04-24T23:03:52.390695+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2977320,"kb_used_data":17788,"kb_used_omap":4,"kb_used_meta":2959483,"kb_avail":280138200,"statfs":{"total":289910292480,"available":286861516800,"internally_reserved":0,"allocated":18214912,"data_stored":10188689,"data_compressed":11512,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":3030510600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.359261"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":500,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.858444+0000","last_change":"2024-04-24T22:28:19.885044+0000","last_active":"2024-04-24T23:03:40.858444+0000","last_peered":"2024-04-24T23:03:40.858444+0000","last_clean":"2024-04-24T23:03:40.858444+0000","last_became_active":"2024-04-24T22:28:19.884405+0000","last_became_peered":"2024-04-24T22:28:19.884405+0000","last_unstale":"2024-04-24T23:03:40.858444+0000","last_undegraded":"2024-04-24T23:03:40.858444+0000","last_fullsized":"2024-04-24T23:03:40.858444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:59:08.229402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":502,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.929185+0000","last_change":"2024-04-24T22:28:19.885719+0000","last_active":"2024-04-24T23:03:39.929185+0000","last_peered":"2024-04-24T23:03:39.929185+0000","last_clean":"2024-04-24T23:03:39.929185+0000","last_became_active":"2024-04-24T22:28:19.885439+0000","last_became_peered":"2024-04-24T22:28:19.885439+0000","last_unstale":"2024-04-24T23:03:39.929185+0000","last_undegraded":"2024-04-24T23:03:39.929185+0000","last_fullsized":"2024-04-24T23:03:39.929185+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:05:50.933126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":501,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:41.430372+0000","last_change":"2024-04-24T22:28:19.883851+0000","last_active":"2024-04-24T23:03:41.430372+0000","last_peered":"2024-04-24T23:03:41.430372+0000","last_clean":"2024-04-24T23:03:41.430372+0000","last_became_active":"2024-04-24T22:28:19.883549+0000","last_became_peered":"2024-04-24T22:28:19.883549+0000","last_unstale":"2024-04-24T23:03:41.430372+0000","last_undegraded":"2024-04-24T23:03:41.430372+0000","last_fullsized":"2024-04-24T23:03:41.430372+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:39:31.099684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":500,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.860533+0000","last_change":"2024-04-24T22:28:19.883883+0000","last_active":"2024-04-24T23:03:40.860533+0000","last_peered":"2024-04-24T23:03:40.860533+0000","last_clean":"2024-04-24T23:03:40.860533+0000","last_became_active":"2024-04-24T22:28:19.883590+0000","last_became_peered":"2024-04-24T22:28:19.883590+0000","last_unstale":"2024-04-24T23:03:40.860533+0000","last_undegraded":"2024-04-24T23:03:40.860533+0000","last_fullsized":"2024-04-24T23:03:40.860533+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:24.775003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":630,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.858703+0000","last_change":"2024-04-24T22:28:13.865092+0000","last_active":"2024-04-24T23:03:40.858703+0000","last_peered":"2024-04-24T23:03:40.858703+0000","last_clean":"2024-04-24T23:03:40.858703+0000","last_became_active":"2024-04-24T22:28:13.864081+0000","last_became_peered":"2024-04-24T22:28:13.864081+0000","last_unstale":"2024-04-24T23:03:40.858703+0000","last_undegraded":"2024-04-24T23:03:40.858703+0000","last_fullsized":"2024-04-24T23:03:40.858703+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:12.854421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:12.854421+0000","last_clean_scrub_stamp":"2024-04-24T22:28:12.854421+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:39:56.775553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":501,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:41.429534+0000","last_change":"2024-04-24T22:28:19.885268+0000","last_active":"2024-04-24T23:03:41.429534+0000","last_peered":"2024-04-24T23:03:41.429534+0000","last_clean":"2024-04-24T23:03:41.429534+0000","last_became_active":"2024-04-24T22:28:19.884948+0000","last_became_peered":"2024-04-24T22:28:19.884948+0000","last_unstale":"2024-04-24T23:03:41.429534+0000","last_undegraded":"2024-04-24T23:03:41.429534+0000","last_fullsized":"2024-04-24T23:03:41.429534+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:00.739359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":500,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.858458+0000","last_change":"2024-04-24T22:28:19.886868+0000","last_active":"2024-04-24T23:03:40.858458+0000","last_peered":"2024-04-24T23:03:40.858458+0000","last_clean":"2024-04-24T23:03:40.858458+0000","last_became_active":"2024-04-24T22:28:19.886592+0000","last_became_peered":"2024-04-24T22:28:19.886592+0000","last_unstale":"2024-04-24T23:03:40.858458+0000","last_undegraded":"2024-04-24T23:03:40.858458+0000","last_fullsized":"2024-04-24T23:03:40.858458+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:50.108365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":500,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.858479+0000","last_change":"2024-04-24T22:28:19.884742+0000","last_active":"2024-04-24T23:03:40.858479+0000","last_peered":"2024-04-24T23:03:40.858479+0000","last_clean":"2024-04-24T23:03:40.858479+0000","last_became_active":"2024-04-24T22:28:19.884428+0000","last_became_peered":"2024-04-24T22:28:19.884428+0000","last_unstale":"2024-04-24T23:03:40.858479+0000","last_undegraded":"2024-04-24T23:03:40.858479+0000","last_fullsized":"2024-04-24T23:03:40.858479+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:12:29.302280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":500,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-04-24T23:03:40.858493+0000","last_change":"2024-04-24T22:28:19.886666+0000","last_active":"2024-04-24T23:03:40.858493+0000","last_peered":"2024-04-24T23:03:40.858493+0000","last_clean":"2024-04-24T23:03:40.858493+0000","last_became_active":"2024-04-24T22:28:19.886369+0000","last_became_peered":"2024-04-24T22:28:19.886369+0000","last_unstale":"2024-04-24T23:03:40.858493+0000","last_undegraded":"2024-04-24T23:03:40.858493+0000","last_fullsized":"2024-04-24T23:03:40.858493+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:28:18.870139+0000","last_clean_scrub_stamp":"2024-04-24T22:28:18.870139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:19:47.445279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11512,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706095,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":884052,"kb_used_data":5892,"kb_used_omap":1,"kb_used_meta":878142,"kb_avail":93487788,"statfs":{"total":96636764160,"available":95731494912,"internally_reserved":0,"allocated":6033408,"data_stored":3002651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":899217752},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 23:03:02 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.82099999999999995,"15min":0.504},"min":{"1min":0.154,"5min":0.154,"15min":0.14699999999999999},"max":{"1min":5.7530000000000001,"5min":12.317,"15min":12.317},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.73099999999999998,"15min":0.46899999999999997},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.16900000000000001},"max":{"1min":3.9910000000000001,"5min":6.1989999999999998,"15min":6.1989999999999998},"last":0.33300000000000002}]},{"osd":1,"last update":"Wed Apr 24 23:03:02 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":0.71599999999999997,"15min":0.48899999999999999},"min":{"1min":0.25700000000000001,"5min":0.185,"15min":0.18099999999999999},"max":{"1min":8.9510000000000005,"5min":8.9510000000000005,"15min":8.9510000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.68899999999999995,"15min":0.46500000000000002},"min":{"1min":0.254,"5min":0.20200000000000001,"15min":0.189},"max":{"1min":3.4689999999999999,"5min":5.4249999999999998,"15min":5.4249999999999998},"last":0.34899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1200968,"kb_used_data":5948,"kb_used_omap":1,"kb_used_meta":1195006,"kb_avail":93170872,"statfs":{"total":96636764160,"available":95406972928,"internally_reserved":0,"allocated":6090752,"data_stored":3593019,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":1223686488},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 23:03:31 2024","interfaces":[{"interface":"back","average":{"1min":1.752,"5min":0.99399999999999999,"15min":0.56599999999999995},"min":{"1min":0.247,"5min":0.156,"15min":0.156},"max":{"1min":13.428000000000001,"5min":13.428000000000001,"15min":13.428000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":1.0980000000000001,"5min":0.83299999999999996,"15min":0.53300000000000003},"min":{"1min":0.20599999999999999,"5min":0.19500000000000001,"15min":0.18099999999999999},"max":{"1min":4.6299999999999999,"5min":7.5279999999999996,"15min":7.5279999999999996},"last":0.434}]},{"osd":2,"last update":"Wed Apr 24 23:03:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":1.0329999999999999,"15min":0.58299999999999996},"min":{"1min":0.249,"5min":0.184,"15min":0.184},"max":{"1min":13.478,"5min":13.478,"15min":13.478},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.60299999999999998,"15min":0.45300000000000001},"min":{"1min":0.27000000000000002,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":2,"5min":6.7130000000000001,"15min":6.7130000000000001},"last":0.39600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738798,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":892300,"kb_used_data":5948,"kb_used_omap":1,"kb_used_meta":886334,"kb_avail":93479540,"statfs":{"total":96636764160,"available":95723048960,"internally_reserved":0,"allocated":6090752,"data_stored":3593019,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":907606360},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 23:03:29 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":0.84999999999999998,"15min":0.55800000000000005},"min":{"1min":0.23699999999999999,"5min":0.155,"15min":0.155},"max":{"1min":14.769,"5min":14.769,"15min":14.769},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61799999999999999,"15min":0.45000000000000001},"min":{"1min":0.26400000000000001,"5min":0.186,"15min":0.186},"max":{"1min":3.8050000000000002,"5min":7.4539999999999997,"15min":7.4539999999999997},"last":0.22600000000000001}]},{"osd":2,"last update":"Wed Apr 24 23:03:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.65500000000000003,"15min":0.53400000000000003},"min":{"1min":0.25800000000000001,"5min":0.182,"15min":0.182},"max":{"1min":4.5670000000000002,"5min":6.4299999999999997,"15min":15.769},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.66700000000000004,"15min":0.44500000000000001},"min":{"1min":0.252,"5min":0.20300000000000001,"15min":0.185},"max":{"1min":2.0619999999999998,"5min":8.3550000000000004,"15min":8.3550000000000004},"last":0.25}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:03:53.720 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-24T22:28:18.870139+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=22, tm_min=28, tm_sec=18, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=3, tm_sec=50, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:03:53.720 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-24T22:28:18.870139+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=22, tm_min=28, tm_sec=18, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=3, tm_sec=50, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:03:53.721 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-24T22:28:18.870139+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=22, tm_min=28, tm_sec=18, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=3, tm_sec=50, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:03:53.721 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-24T22:28:18.870139+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=22, tm_min=28, tm_sec=18, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=3, tm_sec=50, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:03:53.721 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-24T22:28:12.854421+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=22, tm_min=28, tm_sec=12, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=3, tm_sec=50, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:03:53.721 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-24T22:28:18.870139+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=22, tm_min=28, tm_sec=18, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=3, tm_sec=50, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:03:53.721 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-24T22:28:18.870139+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=22, tm_min=28, tm_sec=18, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=3, tm_sec=50, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:03:53.721 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-24T22:28:18.870139+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=22, tm_min=28, tm_sec=18, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=3, tm_sec=50, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:03:53.721 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-24T22:28:18.870139+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=22, tm_min=28, tm_sec=18, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=3, tm_sec=50, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:03:53.721 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-24T23:04:13.722 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-24T23:04:14.045 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-04-24T23:04:14.047 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-04-24T23:04:14.058 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":1585,"stamp":"2024-04-24T23:04:12.394604+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2977300,"kb_used_data":17788,"kb_used_omap":4,"kb_used_meta":2959483,"kb_avail":280138220,"statfs":{"total":289910292480,"available":286861537280,"internally_reserved":0,"allocated":18214912,"data_stored":10188689,"data_compressed":11512,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":3030510600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002273"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":511,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-04-24T23:03:59.796547+0000","last_change":"2024-04-24T23:03:59.796519+0000","last_active":"2024-04-24T23:03:59.796547+0000","last_peered":"2024-04-24T23:03:59.796547+0000","last_clean":"2024-04-24T23:03:59.796547+0000","last_became_active":"2024-04-24T22:28:19.884405+0000","last_became_peered":"2024-04-24T22:28:19.884405+0000","last_unstale":"2024-04-24T23:03:59.796547+0000","last_undegraded":"2024-04-24T23:03:59.796547+0000","last_fullsized":"2024-04-24T23:03:59.796547+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:03:59.796421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:03:59.796421+0000","last_clean_scrub_stamp":"2024-04-24T23:03:59.796421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:03:55.649000+0000","scrub_duration":0.00149442,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":513,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-04-24T23:03:52.775256+0000","last_change":"2024-04-24T23:03:52.775228+0000","last_active":"2024-04-24T23:03:52.775256+0000","last_peered":"2024-04-24T23:03:52.775256+0000","last_clean":"2024-04-24T23:03:52.775256+0000","last_became_active":"2024-04-24T22:28:19.885439+0000","last_became_peered":"2024-04-24T22:28:19.885439+0000","last_unstale":"2024-04-24T23:03:52.775256+0000","last_undegraded":"2024-04-24T23:03:52.775256+0000","last_fullsized":"2024-04-24T23:03:52.775256+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:03:52.775125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:03:52.775125+0000","last_clean_scrub_stamp":"2024-04-24T23:03:52.775125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:11.695686+0000","scrub_duration":0.054714259000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-04-24T23:03:57.718673+0000","last_change":"2024-04-24T23:03:57.718643+0000","last_active":"2024-04-24T23:03:57.718673+0000","last_peered":"2024-04-24T23:03:57.718673+0000","last_clean":"2024-04-24T23:03:57.718673+0000","last_became_active":"2024-04-24T22:28:19.883549+0000","last_became_peered":"2024-04-24T22:28:19.883549+0000","last_unstale":"2024-04-24T23:03:57.718673+0000","last_undegraded":"2024-04-24T23:03:57.718673+0000","last_fullsized":"2024-04-24T23:03:57.718673+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:03:57.718528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:03:57.718528+0000","last_clean_scrub_stamp":"2024-04-24T23:03:57.718528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:45:08.858223+0000","scrub_duration":0.142063044,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":511,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-04-24T23:03:58.011453+0000","last_change":"2024-04-24T23:03:58.011430+0000","last_active":"2024-04-24T23:03:58.011453+0000","last_peered":"2024-04-24T23:03:58.011453+0000","last_clean":"2024-04-24T23:03:58.011453+0000","last_became_active":"2024-04-24T22:28:19.883590+0000","last_became_peered":"2024-04-24T22:28:19.883590+0000","last_unstale":"2024-04-24T23:03:58.011453+0000","last_undegraded":"2024-04-24T23:03:58.011453+0000","last_fullsized":"2024-04-24T23:03:58.011453+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:03:58.011325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:03:58.011325+0000","last_clean_scrub_stamp":"2024-04-24T23:03:58.011325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:09.852306+0000","scrub_duration":0.138925874,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"508'99","reported_seq":643,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-04-24T23:03:54.834139+0000","last_change":"2024-04-24T23:03:54.834117+0000","last_active":"2024-04-24T23:03:54.834139+0000","last_peered":"2024-04-24T23:03:54.834139+0000","last_clean":"2024-04-24T23:03:54.834139+0000","last_became_active":"2024-04-24T22:28:13.864081+0000","last_became_peered":"2024-04-24T22:28:13.864081+0000","last_unstale":"2024-04-24T23:03:54.834139+0000","last_undegraded":"2024-04-24T23:03:54.834139+0000","last_fullsized":"2024-04-24T23:03:54.834139+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'99","last_scrub_stamp":"2024-04-24T23:03:54.834044+0000","last_deep_scrub":"508'99","last_deep_scrub_stamp":"2024-04-24T23:03:54.834044+0000","last_clean_scrub_stamp":"2024-04-24T23:03:54.834044+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:06.374314+0000","scrub_duration":0.0034847939999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":512,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-04-24T23:03:54.683966+0000","last_change":"2024-04-24T23:03:54.683925+0000","last_active":"2024-04-24T23:03:54.683966+0000","last_peered":"2024-04-24T23:03:54.683966+0000","last_clean":"2024-04-24T23:03:54.683966+0000","last_became_active":"2024-04-24T22:28:19.884948+0000","last_became_peered":"2024-04-24T22:28:19.884948+0000","last_unstale":"2024-04-24T23:03:54.683966+0000","last_undegraded":"2024-04-24T23:03:54.683966+0000","last_fullsized":"2024-04-24T23:03:54.683966+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:03:54.683768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:03:54.683768+0000","last_clean_scrub_stamp":"2024-04-24T23:03:54.683768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:10:05.586344+0000","scrub_duration":0.086397142999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":511,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-04-24T23:04:07.906588+0000","last_change":"2024-04-24T23:04:07.906539+0000","last_active":"2024-04-24T23:04:07.906588+0000","last_peered":"2024-04-24T23:04:07.906588+0000","last_clean":"2024-04-24T23:04:07.906588+0000","last_became_active":"2024-04-24T22:28:19.886592+0000","last_became_peered":"2024-04-24T22:28:19.886592+0000","last_unstale":"2024-04-24T23:04:07.906588+0000","last_undegraded":"2024-04-24T23:04:07.906588+0000","last_fullsized":"2024-04-24T23:04:07.906588+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:04:07.906388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:04:07.906388+0000","last_clean_scrub_stamp":"2024-04-24T23:04:07.906388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:14:20.780210+0000","scrub_duration":0.13618666099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":511,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-04-24T23:04:04.873466+0000","last_change":"2024-04-24T23:04:04.873420+0000","last_active":"2024-04-24T23:04:04.873466+0000","last_peered":"2024-04-24T23:04:04.873466+0000","last_clean":"2024-04-24T23:04:04.873466+0000","last_became_active":"2024-04-24T22:28:19.884428+0000","last_became_peered":"2024-04-24T22:28:19.884428+0000","last_unstale":"2024-04-24T23:04:04.873466+0000","last_undegraded":"2024-04-24T23:04:04.873466+0000","last_fullsized":"2024-04-24T23:04:04.873466+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:04:04.873242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:04:04.873242+0000","last_clean_scrub_stamp":"2024-04-24T23:04:04.873242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:45.045197+0000","scrub_duration":0.061457505000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":511,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-04-24T23:04:08.887934+0000","last_change":"2024-04-24T23:04:08.887898+0000","last_active":"2024-04-24T23:04:08.887934+0000","last_peered":"2024-04-24T23:04:08.887934+0000","last_clean":"2024-04-24T23:04:08.887934+0000","last_became_active":"2024-04-24T22:28:19.886369+0000","last_became_peered":"2024-04-24T22:28:19.886369+0000","last_unstale":"2024-04-24T23:04:08.887934+0000","last_undegraded":"2024-04-24T23:04:08.887934+0000","last_fullsized":"2024-04-24T23:04:08.887934+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:04:08.887780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:04:08.887780+0000","last_clean_scrub_stamp":"2024-04-24T23:04:08.887780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:03.019942+0000","scrub_duration":0.159194276,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11512,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706099,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":884044,"kb_used_data":5892,"kb_used_omap":1,"kb_used_meta":878142,"kb_avail":93487796,"statfs":{"total":96636764160,"available":95731503104,"internally_reserved":0,"allocated":6033408,"data_stored":3002651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":899217752},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 23:04:05 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.69299999999999995,"15min":0.50800000000000001},"min":{"1min":0.221,"5min":0.154,"15min":0.14699999999999999},"max":{"1min":0.48499999999999999,"5min":12.317,"15min":12.317},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.61799999999999999,"15min":0.47199999999999998},"min":{"1min":0.27800000000000002,"5min":0.17699999999999999,"15min":0.16900000000000001},"max":{"1min":0.47899999999999998,"5min":6.0919999999999996,"15min":6.1989999999999998},"last":0.32300000000000001}]},{"osd":1,"last update":"Wed Apr 24 23:04:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.68999999999999995,"15min":0.48999999999999999},"min":{"1min":0.25700000000000001,"5min":0.185,"15min":0.18099999999999999},"max":{"1min":0.45600000000000002,"5min":8.9510000000000005,"15min":8.9510000000000005},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.67100000000000004,"15min":0.46700000000000003},"min":{"1min":0.23499999999999999,"5min":0.219,"15min":0.189},"max":{"1min":0.503,"5min":5.4249999999999998,"15min":5.4249999999999998},"last":0.34100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738802,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1200964,"kb_used_data":5948,"kb_used_omap":1,"kb_used_meta":1195006,"kb_avail":93170876,"statfs":{"total":96636764160,"available":95406977024,"internally_reserved":0,"allocated":6090752,"data_stored":3593019,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":1223686488},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 23:03:31 2024","interfaces":[{"interface":"back","average":{"1min":1.752,"5min":0.99399999999999999,"15min":0.56599999999999995},"min":{"1min":0.247,"5min":0.156,"15min":0.156},"max":{"1min":13.428000000000001,"5min":13.428000000000001,"15min":13.428000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":1.0980000000000001,"5min":0.83299999999999996,"15min":0.53300000000000003},"min":{"1min":0.20599999999999999,"5min":0.19500000000000001,"15min":0.18099999999999999},"max":{"1min":4.6299999999999999,"5min":7.5279999999999996,"15min":7.5279999999999996},"last":0.503}]},{"osd":2,"last update":"Wed Apr 24 23:03:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":1.0329999999999999,"15min":0.58299999999999996},"min":{"1min":0.249,"5min":0.184,"15min":0.184},"max":{"1min":13.478,"5min":13.478,"15min":13.478},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.60299999999999998,"15min":0.45300000000000001},"min":{"1min":0.27000000000000002,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":2,"5min":6.7130000000000001,"15min":6.7130000000000001},"last":0.40799999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738802,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":892292,"kb_used_data":5948,"kb_used_omap":1,"kb_used_meta":886334,"kb_avail":93479548,"statfs":{"total":96636764160,"available":95723057152,"internally_reserved":0,"allocated":6090752,"data_stored":3593019,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":907606360},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 23:03:29 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":0.84999999999999998,"15min":0.55800000000000005},"min":{"1min":0.23699999999999999,"5min":0.155,"15min":0.155},"max":{"1min":14.769,"5min":14.769,"15min":14.769},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.61799999999999999,"15min":0.45000000000000001},"min":{"1min":0.26400000000000001,"5min":0.186,"15min":0.186},"max":{"1min":3.8050000000000002,"5min":7.4539999999999997,"15min":7.4539999999999997},"last":0.24099999999999999}]},{"osd":2,"last update":"Wed Apr 24 23:03:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.65500000000000003,"15min":0.53400000000000003},"min":{"1min":0.25800000000000001,"5min":0.182,"15min":0.182},"max":{"1min":4.5670000000000002,"5min":6.4299999999999997,"15min":15.769},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.66700000000000004,"15min":0.44500000000000001},"min":{"1min":0.252,"5min":0.20300000000000001,"15min":0.185},"max":{"1min":2.0619999999999998,"5min":8.3550000000000004,"15min":8.3550000000000004},"last":0.27200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5756,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:04:14.060 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-04-24T23:04:14.385 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-24T23:04:14.385 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-24T23:04:14.385 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-04-24T23:04:14.385 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:04:14.467 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-04-24T23:04:14.468 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-04-24T23:04:14.468 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:04:14.556 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-04-24T23:04:14.556 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-04-24T23:04:14.556 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:04:14.635 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-04-24T23:04:14.635 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-24T23:04:14.635 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-04-24T23:04:14.636 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:04:14.662 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-04-24T23:04:14.663 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-24T23:04:14.664 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-04-24T23:04:14.664 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:04:14.675 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-04-24T23:04:14.676 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-24T23:04:14.676 DEBUG:teuthology.orchestra.run.smithi134:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-04-24T23:04:14.745 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi134.front.sepia.ceph.com 2024-04-24T23:04:14.745 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-04-24T23:04:15.029 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi134.front.sepia.ceph.com 2024-04-24T23:04:15.029 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-04-24T23:04:15.167 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi134.front.sepia.ceph.com 2024-04-24T23:04:15.167 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-04-24T23:04:15.280 INFO:tasks.ceph:Archiving mon data... 2024-04-24T23:04:15.281 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/yuriw-2024-04-24_21:25:53-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7672255/data/cluster2.mon.a.tgz 2024-04-24T23:04:15.282 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-04-24T23:04:15.299 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.SCcLGTcJkB 2024-04-24T23:04:15.299 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.SCcLGTcJkB 2024-04-24T23:04:15.476 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.SCcLGTcJkB 2024-04-24T23:04:15.563 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.SCcLGTcJkB is 436KB 2024-04-24T23:04:15.661 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.SCcLGTcJkB 2024-04-24T23:04:15.678 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-24T23:04:15.678 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T23:04:15.681 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T23:04:15.745 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-24T23:04:15.759 INFO:tasks.ceph.ceph_manager.cluster1:Canceling any pending splits or merges... 2024-04-24T23:04:15.759 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-24T23:04:16.083 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T23:04:16.083 INFO:teuthology.orchestra.run.smithi080.stdout:{"epoch":457,"fsid":"c6f15796-edf2-4aa3-acbd-00f4afb79b95","created":"2024-04-24T22:27:33.193401+0000","modified":"2024-04-24T23:03:41.438855+0000","last_up_change":"2024-04-24T22:27:41.275739+0000","last_in_change":"2024-04-24T22:27:34.169017+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-24T22:27:41.926799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-24T22:27:42.860947+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"30b54047-4212-4616-bdcb-7ed629fbd802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":42679},{"type":"v1","addr":"172.21.15.80:6817","nonce":42679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":42679},{"type":"v1","addr":"172.21.15.80:6819","nonce":42679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":42679},{"type":"v1","addr":"172.21.15.80:6823","nonce":42679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":42679},{"type":"v1","addr":"172.21.15.80:6821","nonce":42679}]},"public_addr":"172.21.15.80:6817/42679","cluster_addr":"172.21.15.80:6819/42679","heartbeat_back_addr":"172.21.15.80:6823/42679","heartbeat_front_addr":"172.21.15.80:6821/42679","state":["exists","up"]},{"osd":1,"uuid":"3137d80f-296e-4fd9-a77f-97031fbe79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":42674},{"type":"v1","addr":"172.21.15.80:6809","nonce":42674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":42674},{"type":"v1","addr":"172.21.15.80:6811","nonce":42674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":42674},{"type":"v1","addr":"172.21.15.80:6815","nonce":42674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":42674},{"type":"v1","addr":"172.21.15.80:6813","nonce":42674}]},"public_addr":"172.21.15.80:6809/42674","cluster_addr":"172.21.15.80:6811/42674","heartbeat_back_addr":"172.21.15.80:6815/42674","heartbeat_front_addr":"172.21.15.80:6813/42674","state":["exists","up"]},{"osd":2,"uuid":"f5cd2fbf-7851-4596-bdee-30b92841e8e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":42673},{"type":"v1","addr":"172.21.15.80:6801","nonce":42673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":42673},{"type":"v1","addr":"172.21.15.80:6803","nonce":42673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":42673},{"type":"v1","addr":"172.21.15.80:6807","nonce":42673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":42673},{"type":"v1","addr":"172.21.15.80:6805","nonce":42673}]},"public_addr":"172.21.15.80:6801/42673","cluster_addr":"172.21.15.80:6803/42673","heartbeat_back_addr":"172.21.15.80:6807/42673","heartbeat_front_addr":"172.21.15.80:6805/42673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:27:39.347037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:27:39.373428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:27:39.360472+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:04:16.097 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-04-24T23:04:16.097 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-24T23:04:16.404 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T23:04:16.406 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-04-24T23:04:16.418 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":1551,"stamp":"2024-04-24T23:04:15.377448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":919572,"kb_used_data":11120,"kb_used_omap":4,"kb_used_meta":908411,"kb_avail":282195948,"statfs":{"total":289910292480,"available":288968650752,"internally_reserved":0,"allocated":11386880,"data_stored":7220438,"data_compressed":11526,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":930212872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002524"},"pg_stats":[{"pgid":"1.5","version":"0'0","reported_seq":455,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.586743+0000","last_change":"2024-04-24T22:27:43.285734+0000","last_active":"2024-04-24T23:03:39.586743+0000","last_peered":"2024-04-24T23:03:39.586743+0000","last_clean":"2024-04-24T23:03:39.586743+0000","last_became_active":"2024-04-24T22:27:43.285451+0000","last_became_peered":"2024-04-24T22:27:43.285451+0000","last_unstale":"2024-04-24T23:03:39.586743+0000","last_undegraded":"2024-04-24T23:03:39.586743+0000","last_fullsized":"2024-04-24T23:03:39.586743+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:05:25.557870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.588224+0000","last_change":"2024-04-24T22:27:43.286584+0000","last_active":"2024-04-24T23:03:39.588224+0000","last_peered":"2024-04-24T23:03:39.588224+0000","last_clean":"2024-04-24T23:03:39.588224+0000","last_became_active":"2024-04-24T22:27:43.285821+0000","last_became_peered":"2024-04-24T22:27:43.285821+0000","last_unstale":"2024-04-24T23:03:39.588224+0000","last_undegraded":"2024-04-24T23:03:39.588224+0000","last_fullsized":"2024-04-24T23:03:39.588224+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:59.162446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.582250+0000","last_change":"2024-04-24T22:27:43.285195+0000","last_active":"2024-04-24T23:03:39.582250+0000","last_peered":"2024-04-24T23:03:39.582250+0000","last_clean":"2024-04-24T23:03:39.582250+0000","last_became_active":"2024-04-24T22:27:43.284939+0000","last_became_peered":"2024-04-24T22:27:43.284939+0000","last_unstale":"2024-04-24T23:03:39.582250+0000","last_undegraded":"2024-04-24T23:03:39.582250+0000","last_fullsized":"2024-04-24T23:03:39.582250+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:17.500091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":581,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.583169+0000","last_change":"2024-04-24T22:27:44.347665+0000","last_active":"2024-04-24T23:03:39.583169+0000","last_peered":"2024-04-24T23:03:39.583169+0000","last_clean":"2024-04-24T23:03:39.583169+0000","last_became_active":"2024-04-24T22:27:44.346752+0000","last_became_peered":"2024-04-24T22:27:44.346752+0000","last_unstale":"2024-04-24T23:03:39.583169+0000","last_undegraded":"2024-04-24T23:03:39.583169+0000","last_fullsized":"2024-04-24T23:03:39.583169+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:43.277831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:43.277831+0000","last_clean_scrub_stamp":"2024-04-24T22:27:43.277831+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:48:56.331694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.444982+0000","last_change":"2024-04-24T22:27:43.285244+0000","last_active":"2024-04-24T23:03:39.444982+0000","last_peered":"2024-04-24T23:03:39.444982+0000","last_clean":"2024-04-24T23:03:39.444982+0000","last_became_active":"2024-04-24T22:27:43.284970+0000","last_became_peered":"2024-04-24T22:27:43.284970+0000","last_unstale":"2024-04-24T23:03:39.444982+0000","last_undegraded":"2024-04-24T23:03:39.444982+0000","last_fullsized":"2024-04-24T23:03:39.444982+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:07:27.079460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.582656+0000","last_change":"2024-04-24T22:27:43.285898+0000","last_active":"2024-04-24T23:03:39.582656+0000","last_peered":"2024-04-24T23:03:39.582656+0000","last_clean":"2024-04-24T23:03:39.582656+0000","last_became_active":"2024-04-24T22:27:43.285646+0000","last_became_peered":"2024-04-24T22:27:43.285646+0000","last_unstale":"2024-04-24T23:03:39.582656+0000","last_undegraded":"2024-04-24T23:03:39.582656+0000","last_fullsized":"2024-04-24T23:03:39.582656+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:30.713659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":455,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.582295+0000","last_change":"2024-04-24T22:27:43.286455+0000","last_active":"2024-04-24T23:03:39.582295+0000","last_peered":"2024-04-24T23:03:39.582295+0000","last_clean":"2024-04-24T23:03:39.582295+0000","last_became_active":"2024-04-24T22:27:43.286197+0000","last_became_peered":"2024-04-24T22:27:43.286197+0000","last_unstale":"2024-04-24T23:03:39.582295+0000","last_undegraded":"2024-04-24T23:03:39.582295+0000","last_fullsized":"2024-04-24T23:03:39.582295+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:04:48.931409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.582535+0000","last_change":"2024-04-24T22:27:43.286377+0000","last_active":"2024-04-24T23:03:39.582535+0000","last_peered":"2024-04-24T23:03:39.582535+0000","last_clean":"2024-04-24T23:03:39.582535+0000","last_became_active":"2024-04-24T22:27:43.286126+0000","last_became_peered":"2024-04-24T22:27:43.286126+0000","last_unstale":"2024-04-24T23:03:39.582535+0000","last_undegraded":"2024-04-24T23:03:39.582535+0000","last_fullsized":"2024-04-24T23:03:39.582535+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:31:54.138109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.593110+0000","last_change":"2024-04-24T22:27:43.285088+0000","last_active":"2024-04-24T23:03:39.593110+0000","last_peered":"2024-04-24T23:03:39.593110+0000","last_clean":"2024-04-24T23:03:39.593110+0000","last_became_active":"2024-04-24T22:27:43.284828+0000","last_became_peered":"2024-04-24T22:27:43.284828+0000","last_unstale":"2024-04-24T23:03:39.593110+0000","last_undegraded":"2024-04-24T23:03:39.593110+0000","last_fullsized":"2024-04-24T23:03:39.593110+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:49:43.635606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11526,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738809,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305176,"kb_used_data":3724,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94066664,"statfs":{"total":96636764160,"available":96324263936,"internally_reserved":0,"allocated":3813376,"data_stored":2603376,"data_compressed":5763,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 23:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.29399999999999998,"15min":0.29799999999999999},"min":{"1min":0.22700000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":1.361},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.32400000000000001,"15min":0.318},"min":{"1min":0.23200000000000001,"5min":0.216,"15min":0.19},"max":{"1min":0.51100000000000001,"5min":0.51700000000000002,"15min":1.264},"last":0.45700000000000002}]},{"osd":1,"last update":"Wed Apr 24 23:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.32400000000000001,"15min":0.33800000000000002},"min":{"1min":0.26100000000000001,"5min":0.223,"15min":0.20000000000000001},"max":{"1min":0.441,"5min":0.47099999999999997,"15min":6.2469999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.35199999999999998,"15min":0.34000000000000002},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.22500000000000001},"max":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":1.3859999999999999},"last":0.499}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313312,"kb_used_data":3668,"kb_used_omap":1,"kb_used_meta":309630,"kb_avail":94058528,"statfs":{"total":96636764160,"available":96315932672,"internally_reserved":0,"allocated":3756032,"data_stored":2013008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317061464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 23:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.29299999999999998,"5min":0.29099999999999998,"15min":0.312},"min":{"1min":0.22,"5min":0.16900000000000001,"15min":0.14399999999999999},"max":{"1min":0.41699999999999998,"5min":0.70699999999999996,"15min":3.0139999999999998},"last":0.307},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.33500000000000002,"15min":0.317},"min":{"1min":0.26400000000000001,"5min":0.23100000000000001,"15min":0.19600000000000001},"max":{"1min":0.50600000000000001,"5min":0.67500000000000004,"15min":0.79100000000000004},"last":0.34300000000000003}]},{"osd":2,"last update":"Wed Apr 24 23:04:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.33900000000000002,"15min":0.33300000000000002},"min":{"1min":0.28100000000000003,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":0.46500000000000002,"5min":0.50700000000000001,"15min":1.415},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.317,"15min":0.32700000000000001},"min":{"1min":0.26500000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47299999999999998,"5min":0.47999999999999998,"15min":1.9159999999999999},"last":0.32000000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738810,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301084,"kb_used_data":3728,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94070756,"statfs":{"total":96636764160,"available":96328454144,"internally_reserved":0,"allocated":3817472,"data_stored":2604054,"data_compressed":5763,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 23:03:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.308,"15min":0.29599999999999999},"min":{"1min":0.22600000000000001,"5min":0.17199999999999999,"15min":0.158},"max":{"1min":0.41699999999999998,"5min":0.45700000000000002,"15min":0.46600000000000003},"last":0.373},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.34399999999999997,"15min":0.314},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.17000000000000001},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.85199999999999998},"last":0.35899999999999999}]},{"osd":2,"last update":"Wed Apr 24 23:03:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.313,"15min":0.316},"min":{"1min":0.26300000000000001,"5min":0.20799999999999999,"15min":0.185},"max":{"1min":0.46200000000000002,"5min":0.46500000000000002,"15min":0.50700000000000001},"last":0.316},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.31900000000000001,"15min":0.33100000000000002},"min":{"1min":0.25800000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.499,"5min":0.499,"15min":0.54800000000000004},"last":0.29099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5763,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5763,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:04:16.419 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-24T23:04:16.727 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T23:04:16.729 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-04-24T23:04:16.740 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":1551,"stamp":"2024-04-24T23:04:15.377448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":919572,"kb_used_data":11120,"kb_used_omap":4,"kb_used_meta":908411,"kb_avail":282195948,"statfs":{"total":289910292480,"available":288968650752,"internally_reserved":0,"allocated":11386880,"data_stored":7220438,"data_compressed":11526,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":930212872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002524"},"pg_stats":[{"pgid":"1.5","version":"0'0","reported_seq":455,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.586743+0000","last_change":"2024-04-24T22:27:43.285734+0000","last_active":"2024-04-24T23:03:39.586743+0000","last_peered":"2024-04-24T23:03:39.586743+0000","last_clean":"2024-04-24T23:03:39.586743+0000","last_became_active":"2024-04-24T22:27:43.285451+0000","last_became_peered":"2024-04-24T22:27:43.285451+0000","last_unstale":"2024-04-24T23:03:39.586743+0000","last_undegraded":"2024-04-24T23:03:39.586743+0000","last_fullsized":"2024-04-24T23:03:39.586743+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:05:25.557870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.588224+0000","last_change":"2024-04-24T22:27:43.286584+0000","last_active":"2024-04-24T23:03:39.588224+0000","last_peered":"2024-04-24T23:03:39.588224+0000","last_clean":"2024-04-24T23:03:39.588224+0000","last_became_active":"2024-04-24T22:27:43.285821+0000","last_became_peered":"2024-04-24T22:27:43.285821+0000","last_unstale":"2024-04-24T23:03:39.588224+0000","last_undegraded":"2024-04-24T23:03:39.588224+0000","last_fullsized":"2024-04-24T23:03:39.588224+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:59.162446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.582250+0000","last_change":"2024-04-24T22:27:43.285195+0000","last_active":"2024-04-24T23:03:39.582250+0000","last_peered":"2024-04-24T23:03:39.582250+0000","last_clean":"2024-04-24T23:03:39.582250+0000","last_became_active":"2024-04-24T22:27:43.284939+0000","last_became_peered":"2024-04-24T22:27:43.284939+0000","last_unstale":"2024-04-24T23:03:39.582250+0000","last_undegraded":"2024-04-24T23:03:39.582250+0000","last_fullsized":"2024-04-24T23:03:39.582250+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:17.500091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":581,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.583169+0000","last_change":"2024-04-24T22:27:44.347665+0000","last_active":"2024-04-24T23:03:39.583169+0000","last_peered":"2024-04-24T23:03:39.583169+0000","last_clean":"2024-04-24T23:03:39.583169+0000","last_became_active":"2024-04-24T22:27:44.346752+0000","last_became_peered":"2024-04-24T22:27:44.346752+0000","last_unstale":"2024-04-24T23:03:39.583169+0000","last_undegraded":"2024-04-24T23:03:39.583169+0000","last_fullsized":"2024-04-24T23:03:39.583169+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:43.277831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:43.277831+0000","last_clean_scrub_stamp":"2024-04-24T22:27:43.277831+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:48:56.331694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.444982+0000","last_change":"2024-04-24T22:27:43.285244+0000","last_active":"2024-04-24T23:03:39.444982+0000","last_peered":"2024-04-24T23:03:39.444982+0000","last_clean":"2024-04-24T23:03:39.444982+0000","last_became_active":"2024-04-24T22:27:43.284970+0000","last_became_peered":"2024-04-24T22:27:43.284970+0000","last_unstale":"2024-04-24T23:03:39.444982+0000","last_undegraded":"2024-04-24T23:03:39.444982+0000","last_fullsized":"2024-04-24T23:03:39.444982+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:07:27.079460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.582656+0000","last_change":"2024-04-24T22:27:43.285898+0000","last_active":"2024-04-24T23:03:39.582656+0000","last_peered":"2024-04-24T23:03:39.582656+0000","last_clean":"2024-04-24T23:03:39.582656+0000","last_became_active":"2024-04-24T22:27:43.285646+0000","last_became_peered":"2024-04-24T22:27:43.285646+0000","last_unstale":"2024-04-24T23:03:39.582656+0000","last_undegraded":"2024-04-24T23:03:39.582656+0000","last_fullsized":"2024-04-24T23:03:39.582656+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:30.713659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":455,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.582295+0000","last_change":"2024-04-24T22:27:43.286455+0000","last_active":"2024-04-24T23:03:39.582295+0000","last_peered":"2024-04-24T23:03:39.582295+0000","last_clean":"2024-04-24T23:03:39.582295+0000","last_became_active":"2024-04-24T22:27:43.286197+0000","last_became_peered":"2024-04-24T22:27:43.286197+0000","last_unstale":"2024-04-24T23:03:39.582295+0000","last_undegraded":"2024-04-24T23:03:39.582295+0000","last_fullsized":"2024-04-24T23:03:39.582295+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:04:48.931409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.582535+0000","last_change":"2024-04-24T22:27:43.286377+0000","last_active":"2024-04-24T23:03:39.582535+0000","last_peered":"2024-04-24T23:03:39.582535+0000","last_clean":"2024-04-24T23:03:39.582535+0000","last_became_active":"2024-04-24T22:27:43.286126+0000","last_became_peered":"2024-04-24T22:27:43.286126+0000","last_unstale":"2024-04-24T23:03:39.582535+0000","last_undegraded":"2024-04-24T23:03:39.582535+0000","last_fullsized":"2024-04-24T23:03:39.582535+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:31:54.138109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.593110+0000","last_change":"2024-04-24T22:27:43.285088+0000","last_active":"2024-04-24T23:03:39.593110+0000","last_peered":"2024-04-24T23:03:39.593110+0000","last_clean":"2024-04-24T23:03:39.593110+0000","last_became_active":"2024-04-24T22:27:43.284828+0000","last_became_peered":"2024-04-24T22:27:43.284828+0000","last_unstale":"2024-04-24T23:03:39.593110+0000","last_undegraded":"2024-04-24T23:03:39.593110+0000","last_fullsized":"2024-04-24T23:03:39.593110+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:49:43.635606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11526,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738809,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305176,"kb_used_data":3724,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94066664,"statfs":{"total":96636764160,"available":96324263936,"internally_reserved":0,"allocated":3813376,"data_stored":2603376,"data_compressed":5763,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 23:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.29399999999999998,"15min":0.29799999999999999},"min":{"1min":0.22700000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":1.361},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.32400000000000001,"15min":0.318},"min":{"1min":0.23200000000000001,"5min":0.216,"15min":0.19},"max":{"1min":0.51100000000000001,"5min":0.51700000000000002,"15min":1.264},"last":0.45700000000000002}]},{"osd":1,"last update":"Wed Apr 24 23:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.32400000000000001,"15min":0.33800000000000002},"min":{"1min":0.26100000000000001,"5min":0.223,"15min":0.20000000000000001},"max":{"1min":0.441,"5min":0.47099999999999997,"15min":6.2469999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.35199999999999998,"15min":0.34000000000000002},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.22500000000000001},"max":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":1.3859999999999999},"last":0.499}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313312,"kb_used_data":3668,"kb_used_omap":1,"kb_used_meta":309630,"kb_avail":94058528,"statfs":{"total":96636764160,"available":96315932672,"internally_reserved":0,"allocated":3756032,"data_stored":2013008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317061464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 23:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.29299999999999998,"5min":0.29099999999999998,"15min":0.312},"min":{"1min":0.22,"5min":0.16900000000000001,"15min":0.14399999999999999},"max":{"1min":0.41699999999999998,"5min":0.70699999999999996,"15min":3.0139999999999998},"last":0.307},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.33500000000000002,"15min":0.317},"min":{"1min":0.26400000000000001,"5min":0.23100000000000001,"15min":0.19600000000000001},"max":{"1min":0.50600000000000001,"5min":0.67500000000000004,"15min":0.79100000000000004},"last":0.34300000000000003}]},{"osd":2,"last update":"Wed Apr 24 23:04:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.33900000000000002,"15min":0.33300000000000002},"min":{"1min":0.28100000000000003,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":0.46500000000000002,"5min":0.50700000000000001,"15min":1.415},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.317,"15min":0.32700000000000001},"min":{"1min":0.26500000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47299999999999998,"5min":0.47999999999999998,"15min":1.9159999999999999},"last":0.32000000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738810,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301084,"kb_used_data":3728,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94070756,"statfs":{"total":96636764160,"available":96328454144,"internally_reserved":0,"allocated":3817472,"data_stored":2604054,"data_compressed":5763,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 23:03:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.308,"15min":0.29599999999999999},"min":{"1min":0.22600000000000001,"5min":0.17199999999999999,"15min":0.158},"max":{"1min":0.41699999999999998,"5min":0.45700000000000002,"15min":0.46600000000000003},"last":0.373},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.34399999999999997,"15min":0.314},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.17000000000000001},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.85199999999999998},"last":0.35899999999999999}]},{"osd":2,"last update":"Wed Apr 24 23:03:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.313,"15min":0.316},"min":{"1min":0.26300000000000001,"5min":0.20799999999999999,"15min":0.185},"max":{"1min":0.46200000000000002,"5min":0.46500000000000002,"15min":0.50700000000000001},"last":0.316},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.31900000000000001,"15min":0.33100000000000002},"min":{"1min":0.25800000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.499,"5min":0.499,"15min":0.54800000000000004},"last":0.29099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5763,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5763,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:04:16.741 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-04-24T23:04:16.741 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-24T23:04:17.061 INFO:teuthology.orchestra.run.smithi080.stderr:2024-04-24T23:04:17.061+0000 7fbcc5dbd700 0 --2- 172.21.15.80:0/1990781052 >> [v2:172.21.15.80:3300/0,v1:172.21.15.80:6789/0] conn(0x7fbcc00a4040 0x7fbcc00aee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-24T23:04:17.249 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T23:04:17.252 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-04-24T23:04:17.263 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":1551,"stamp":"2024-04-24T23:04:15.377448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":919572,"kb_used_data":11120,"kb_used_omap":4,"kb_used_meta":908411,"kb_avail":282195948,"statfs":{"total":289910292480,"available":288968650752,"internally_reserved":0,"allocated":11386880,"data_stored":7220438,"data_compressed":11526,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":930212872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002524"},"pg_stats":[{"pgid":"1.5","version":"0'0","reported_seq":455,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.586743+0000","last_change":"2024-04-24T22:27:43.285734+0000","last_active":"2024-04-24T23:03:39.586743+0000","last_peered":"2024-04-24T23:03:39.586743+0000","last_clean":"2024-04-24T23:03:39.586743+0000","last_became_active":"2024-04-24T22:27:43.285451+0000","last_became_peered":"2024-04-24T22:27:43.285451+0000","last_unstale":"2024-04-24T23:03:39.586743+0000","last_undegraded":"2024-04-24T23:03:39.586743+0000","last_fullsized":"2024-04-24T23:03:39.586743+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:05:25.557870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.588224+0000","last_change":"2024-04-24T22:27:43.286584+0000","last_active":"2024-04-24T23:03:39.588224+0000","last_peered":"2024-04-24T23:03:39.588224+0000","last_clean":"2024-04-24T23:03:39.588224+0000","last_became_active":"2024-04-24T22:27:43.285821+0000","last_became_peered":"2024-04-24T22:27:43.285821+0000","last_unstale":"2024-04-24T23:03:39.588224+0000","last_undegraded":"2024-04-24T23:03:39.588224+0000","last_fullsized":"2024-04-24T23:03:39.588224+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:59.162446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.582250+0000","last_change":"2024-04-24T22:27:43.285195+0000","last_active":"2024-04-24T23:03:39.582250+0000","last_peered":"2024-04-24T23:03:39.582250+0000","last_clean":"2024-04-24T23:03:39.582250+0000","last_became_active":"2024-04-24T22:27:43.284939+0000","last_became_peered":"2024-04-24T22:27:43.284939+0000","last_unstale":"2024-04-24T23:03:39.582250+0000","last_undegraded":"2024-04-24T23:03:39.582250+0000","last_fullsized":"2024-04-24T23:03:39.582250+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:17.500091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":581,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.583169+0000","last_change":"2024-04-24T22:27:44.347665+0000","last_active":"2024-04-24T23:03:39.583169+0000","last_peered":"2024-04-24T23:03:39.583169+0000","last_clean":"2024-04-24T23:03:39.583169+0000","last_became_active":"2024-04-24T22:27:44.346752+0000","last_became_peered":"2024-04-24T22:27:44.346752+0000","last_unstale":"2024-04-24T23:03:39.583169+0000","last_undegraded":"2024-04-24T23:03:39.583169+0000","last_fullsized":"2024-04-24T23:03:39.583169+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:43.277831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:43.277831+0000","last_clean_scrub_stamp":"2024-04-24T22:27:43.277831+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:48:56.331694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.444982+0000","last_change":"2024-04-24T22:27:43.285244+0000","last_active":"2024-04-24T23:03:39.444982+0000","last_peered":"2024-04-24T23:03:39.444982+0000","last_clean":"2024-04-24T23:03:39.444982+0000","last_became_active":"2024-04-24T22:27:43.284970+0000","last_became_peered":"2024-04-24T22:27:43.284970+0000","last_unstale":"2024-04-24T23:03:39.444982+0000","last_undegraded":"2024-04-24T23:03:39.444982+0000","last_fullsized":"2024-04-24T23:03:39.444982+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:07:27.079460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.582656+0000","last_change":"2024-04-24T22:27:43.285898+0000","last_active":"2024-04-24T23:03:39.582656+0000","last_peered":"2024-04-24T23:03:39.582656+0000","last_clean":"2024-04-24T23:03:39.582656+0000","last_became_active":"2024-04-24T22:27:43.285646+0000","last_became_peered":"2024-04-24T22:27:43.285646+0000","last_unstale":"2024-04-24T23:03:39.582656+0000","last_undegraded":"2024-04-24T23:03:39.582656+0000","last_fullsized":"2024-04-24T23:03:39.582656+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:30.713659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":455,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.582295+0000","last_change":"2024-04-24T22:27:43.286455+0000","last_active":"2024-04-24T23:03:39.582295+0000","last_peered":"2024-04-24T23:03:39.582295+0000","last_clean":"2024-04-24T23:03:39.582295+0000","last_became_active":"2024-04-24T22:27:43.286197+0000","last_became_peered":"2024-04-24T22:27:43.286197+0000","last_unstale":"2024-04-24T23:03:39.582295+0000","last_undegraded":"2024-04-24T23:03:39.582295+0000","last_fullsized":"2024-04-24T23:03:39.582295+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:04:48.931409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.582535+0000","last_change":"2024-04-24T22:27:43.286377+0000","last_active":"2024-04-24T23:03:39.582535+0000","last_peered":"2024-04-24T23:03:39.582535+0000","last_clean":"2024-04-24T23:03:39.582535+0000","last_became_active":"2024-04-24T22:27:43.286126+0000","last_became_peered":"2024-04-24T22:27:43.286126+0000","last_unstale":"2024-04-24T23:03:39.582535+0000","last_undegraded":"2024-04-24T23:03:39.582535+0000","last_fullsized":"2024-04-24T23:03:39.582535+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:31:54.138109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.593110+0000","last_change":"2024-04-24T22:27:43.285088+0000","last_active":"2024-04-24T23:03:39.593110+0000","last_peered":"2024-04-24T23:03:39.593110+0000","last_clean":"2024-04-24T23:03:39.593110+0000","last_became_active":"2024-04-24T22:27:43.284828+0000","last_became_peered":"2024-04-24T22:27:43.284828+0000","last_unstale":"2024-04-24T23:03:39.593110+0000","last_undegraded":"2024-04-24T23:03:39.593110+0000","last_fullsized":"2024-04-24T23:03:39.593110+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:49:43.635606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11526,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738809,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305176,"kb_used_data":3724,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94066664,"statfs":{"total":96636764160,"available":96324263936,"internally_reserved":0,"allocated":3813376,"data_stored":2603376,"data_compressed":5763,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 23:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.29399999999999998,"15min":0.29799999999999999},"min":{"1min":0.22700000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":1.361},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.32400000000000001,"15min":0.318},"min":{"1min":0.23200000000000001,"5min":0.216,"15min":0.19},"max":{"1min":0.51100000000000001,"5min":0.51700000000000002,"15min":1.264},"last":0.45700000000000002}]},{"osd":1,"last update":"Wed Apr 24 23:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.32400000000000001,"15min":0.33800000000000002},"min":{"1min":0.26100000000000001,"5min":0.223,"15min":0.20000000000000001},"max":{"1min":0.441,"5min":0.47099999999999997,"15min":6.2469999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.35199999999999998,"15min":0.34000000000000002},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.22500000000000001},"max":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":1.3859999999999999},"last":0.499}]}]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313312,"kb_used_data":3668,"kb_used_omap":1,"kb_used_meta":309630,"kb_avail":94058528,"statfs":{"total":96636764160,"available":96315932672,"internally_reserved":0,"allocated":3756032,"data_stored":2013008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317061464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 23:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.29299999999999998,"5min":0.29099999999999998,"15min":0.312},"min":{"1min":0.22,"5min":0.16900000000000001,"15min":0.14399999999999999},"max":{"1min":0.41699999999999998,"5min":0.70699999999999996,"15min":3.0139999999999998},"last":0.307},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.33500000000000002,"15min":0.317},"min":{"1min":0.26400000000000001,"5min":0.23100000000000001,"15min":0.19600000000000001},"max":{"1min":0.50600000000000001,"5min":0.67500000000000004,"15min":0.79100000000000004},"last":0.34300000000000003}]},{"osd":2,"last update":"Wed Apr 24 23:04:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.33900000000000002,"15min":0.33300000000000002},"min":{"1min":0.28100000000000003,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":0.46500000000000002,"5min":0.50700000000000001,"15min":1.415},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.317,"15min":0.32700000000000001},"min":{"1min":0.26500000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47299999999999998,"5min":0.47999999999999998,"15min":1.9159999999999999},"last":0.32000000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738810,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301084,"kb_used_data":3728,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94070756,"statfs":{"total":96636764160,"available":96328454144,"internally_reserved":0,"allocated":3817472,"data_stored":2604054,"data_compressed":5763,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 23:03:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.308,"15min":0.29599999999999999},"min":{"1min":0.22600000000000001,"5min":0.17199999999999999,"15min":0.158},"max":{"1min":0.41699999999999998,"5min":0.45700000000000002,"15min":0.46600000000000003},"last":0.373},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.34399999999999997,"15min":0.314},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.17000000000000001},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.85199999999999998},"last":0.35899999999999999}]},{"osd":2,"last update":"Wed Apr 24 23:03:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.313,"15min":0.316},"min":{"1min":0.26300000000000001,"5min":0.20799999999999999,"15min":0.185},"max":{"1min":0.46200000000000002,"5min":0.46500000000000002,"15min":0.50700000000000001},"last":0.316},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.31900000000000001,"15min":0.33100000000000002},"min":{"1min":0.25800000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.499,"5min":0.499,"15min":0.54800000000000004},"last":0.29099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5763,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5763,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:04:17.264 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-24T23:04:17.577 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T23:04:17.577 INFO:teuthology.orchestra.run.smithi080.stdout:{"epoch":457,"fsid":"c6f15796-edf2-4aa3-acbd-00f4afb79b95","created":"2024-04-24T22:27:33.193401+0000","modified":"2024-04-24T23:03:41.438855+0000","last_up_change":"2024-04-24T22:27:41.275739+0000","last_in_change":"2024-04-24T22:27:34.169017+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-24T22:27:41.926799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-24T22:27:42.860947+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"30b54047-4212-4616-bdcb-7ed629fbd802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6816","nonce":42679},{"type":"v1","addr":"172.21.15.80:6817","nonce":42679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6818","nonce":42679},{"type":"v1","addr":"172.21.15.80:6819","nonce":42679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6822","nonce":42679},{"type":"v1","addr":"172.21.15.80:6823","nonce":42679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6820","nonce":42679},{"type":"v1","addr":"172.21.15.80:6821","nonce":42679}]},"public_addr":"172.21.15.80:6817/42679","cluster_addr":"172.21.15.80:6819/42679","heartbeat_back_addr":"172.21.15.80:6823/42679","heartbeat_front_addr":"172.21.15.80:6821/42679","state":["exists","up"]},{"osd":1,"uuid":"3137d80f-296e-4fd9-a77f-97031fbe79b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6808","nonce":42674},{"type":"v1","addr":"172.21.15.80:6809","nonce":42674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6810","nonce":42674},{"type":"v1","addr":"172.21.15.80:6811","nonce":42674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6814","nonce":42674},{"type":"v1","addr":"172.21.15.80:6815","nonce":42674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6812","nonce":42674},{"type":"v1","addr":"172.21.15.80:6813","nonce":42674}]},"public_addr":"172.21.15.80:6809/42674","cluster_addr":"172.21.15.80:6811/42674","heartbeat_back_addr":"172.21.15.80:6815/42674","heartbeat_front_addr":"172.21.15.80:6813/42674","state":["exists","up"]},{"osd":2,"uuid":"f5cd2fbf-7851-4596-bdee-30b92841e8e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6800","nonce":42673},{"type":"v1","addr":"172.21.15.80:6801","nonce":42673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6802","nonce":42673},{"type":"v1","addr":"172.21.15.80:6803","nonce":42673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6806","nonce":42673},{"type":"v1","addr":"172.21.15.80:6807","nonce":42673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.80:6804","nonce":42673},{"type":"v1","addr":"172.21.15.80:6805","nonce":42673}]},"public_addr":"172.21.15.80:6801/42673","cluster_addr":"172.21.15.80:6803/42673","heartbeat_back_addr":"172.21.15.80:6807/42673","heartbeat_front_addr":"172.21.15.80:6805/42673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:27:39.347037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:27:39.373428+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T22:27:39.360472+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:04:18.592 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-04-24T23:04:18.592 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-24T23:04:18.748 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-04-24T23:04:18.748 INFO:teuthology.orchestra.run.smithi080.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T23:04:18.749 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-04-24T23:04:18.760 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-04-24T23:04:19.074 INFO:teuthology.orchestra.run.smithi080.stderr:instructed osd(s) 0 to deep-scrub 2024-04-24T23:04:19.085 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-04-24T23:04:19.085 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-24T23:04:19.224 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-04-24T23:04:19.224 INFO:teuthology.orchestra.run.smithi080.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T23:04:19.224 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-04-24T23:04:19.236 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-04-24T23:04:19.550 INFO:teuthology.orchestra.run.smithi080.stderr:instructed osd(s) 1 to deep-scrub 2024-04-24T23:04:19.561 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-04-24T23:04:19.561 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-24T23:04:19.700 INFO:teuthology.orchestra.run.smithi080.stdout:{ 2024-04-24T23:04:19.700 INFO:teuthology.orchestra.run.smithi080.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T23:04:19.700 INFO:teuthology.orchestra.run.smithi080.stdout:} 2024-04-24T23:04:19.712 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-04-24T23:04:20.026 INFO:teuthology.orchestra.run.smithi080.stderr:instructed osd(s) 2 to deep-scrub 2024-04-24T23:04:20.038 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-24T23:04:20.343 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T23:04:20.345 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-04-24T23:04:20.356 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":1553,"stamp":"2024-04-24T23:04:19.378053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":919572,"kb_used_data":11120,"kb_used_omap":4,"kb_used_meta":908411,"kb_avail":282195948,"statfs":{"total":289910292480,"available":288968650752,"internally_reserved":0,"allocated":11386880,"data_stored":7220438,"data_compressed":11526,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":930212872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002173"},"pg_stats":[{"pgid":"1.5","version":"0'0","reported_seq":455,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.586743+0000","last_change":"2024-04-24T22:27:43.285734+0000","last_active":"2024-04-24T23:03:39.586743+0000","last_peered":"2024-04-24T23:03:39.586743+0000","last_clean":"2024-04-24T23:03:39.586743+0000","last_became_active":"2024-04-24T22:27:43.285451+0000","last_became_peered":"2024-04-24T22:27:43.285451+0000","last_unstale":"2024-04-24T23:03:39.586743+0000","last_undegraded":"2024-04-24T23:03:39.586743+0000","last_fullsized":"2024-04-24T23:03:39.586743+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:05:25.557870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.588224+0000","last_change":"2024-04-24T22:27:43.286584+0000","last_active":"2024-04-24T23:03:39.588224+0000","last_peered":"2024-04-24T23:03:39.588224+0000","last_clean":"2024-04-24T23:03:39.588224+0000","last_became_active":"2024-04-24T22:27:43.285821+0000","last_became_peered":"2024-04-24T22:27:43.285821+0000","last_unstale":"2024-04-24T23:03:39.588224+0000","last_undegraded":"2024-04-24T23:03:39.588224+0000","last_fullsized":"2024-04-24T23:03:39.588224+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:59.162446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.582250+0000","last_change":"2024-04-24T22:27:43.285195+0000","last_active":"2024-04-24T23:03:39.582250+0000","last_peered":"2024-04-24T23:03:39.582250+0000","last_clean":"2024-04-24T23:03:39.582250+0000","last_became_active":"2024-04-24T22:27:43.284939+0000","last_became_peered":"2024-04-24T22:27:43.284939+0000","last_unstale":"2024-04-24T23:03:39.582250+0000","last_undegraded":"2024-04-24T23:03:39.582250+0000","last_fullsized":"2024-04-24T23:03:39.582250+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:17.500091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":581,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.583169+0000","last_change":"2024-04-24T22:27:44.347665+0000","last_active":"2024-04-24T23:03:39.583169+0000","last_peered":"2024-04-24T23:03:39.583169+0000","last_clean":"2024-04-24T23:03:39.583169+0000","last_became_active":"2024-04-24T22:27:44.346752+0000","last_became_peered":"2024-04-24T22:27:44.346752+0000","last_unstale":"2024-04-24T23:03:39.583169+0000","last_undegraded":"2024-04-24T23:03:39.583169+0000","last_fullsized":"2024-04-24T23:03:39.583169+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:43.277831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:43.277831+0000","last_clean_scrub_stamp":"2024-04-24T22:27:43.277831+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:48:56.331694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.444982+0000","last_change":"2024-04-24T22:27:43.285244+0000","last_active":"2024-04-24T23:03:39.444982+0000","last_peered":"2024-04-24T23:03:39.444982+0000","last_clean":"2024-04-24T23:03:39.444982+0000","last_became_active":"2024-04-24T22:27:43.284970+0000","last_became_peered":"2024-04-24T22:27:43.284970+0000","last_unstale":"2024-04-24T23:03:39.444982+0000","last_undegraded":"2024-04-24T23:03:39.444982+0000","last_fullsized":"2024-04-24T23:03:39.444982+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:07:27.079460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.582656+0000","last_change":"2024-04-24T22:27:43.285898+0000","last_active":"2024-04-24T23:03:39.582656+0000","last_peered":"2024-04-24T23:03:39.582656+0000","last_clean":"2024-04-24T23:03:39.582656+0000","last_became_active":"2024-04-24T22:27:43.285646+0000","last_became_peered":"2024-04-24T22:27:43.285646+0000","last_unstale":"2024-04-24T23:03:39.582656+0000","last_undegraded":"2024-04-24T23:03:39.582656+0000","last_fullsized":"2024-04-24T23:03:39.582656+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:30.713659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":455,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.582295+0000","last_change":"2024-04-24T22:27:43.286455+0000","last_active":"2024-04-24T23:03:39.582295+0000","last_peered":"2024-04-24T23:03:39.582295+0000","last_clean":"2024-04-24T23:03:39.582295+0000","last_became_active":"2024-04-24T22:27:43.286197+0000","last_became_peered":"2024-04-24T22:27:43.286197+0000","last_unstale":"2024-04-24T23:03:39.582295+0000","last_undegraded":"2024-04-24T23:03:39.582295+0000","last_fullsized":"2024-04-24T23:03:39.582295+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:04:48.931409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.582535+0000","last_change":"2024-04-24T22:27:43.286377+0000","last_active":"2024-04-24T23:03:39.582535+0000","last_peered":"2024-04-24T23:03:39.582535+0000","last_clean":"2024-04-24T23:03:39.582535+0000","last_became_active":"2024-04-24T22:27:43.286126+0000","last_became_peered":"2024-04-24T22:27:43.286126+0000","last_unstale":"2024-04-24T23:03:39.582535+0000","last_undegraded":"2024-04-24T23:03:39.582535+0000","last_fullsized":"2024-04-24T23:03:39.582535+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:31:54.138109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-04-24T23:03:39.593110+0000","last_change":"2024-04-24T22:27:43.285088+0000","last_active":"2024-04-24T23:03:39.593110+0000","last_peered":"2024-04-24T23:03:39.593110+0000","last_clean":"2024-04-24T23:03:39.593110+0000","last_became_active":"2024-04-24T22:27:43.284828+0000","last_became_peered":"2024-04-24T22:27:43.284828+0000","last_unstale":"2024-04-24T23:03:39.593110+0000","last_undegraded":"2024-04-24T23:03:39.593110+0000","last_fullsized":"2024-04-24T23:03:39.593110+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T22:27:42.276465+0000","last_clean_scrub_stamp":"2024-04-24T22:27:42.276465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:49:43.635606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11526,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738810,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305176,"kb_used_data":3724,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94066664,"statfs":{"total":96636764160,"available":96324263936,"internally_reserved":0,"allocated":3813376,"data_stored":2603376,"data_compressed":5763,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 23:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.29399999999999998,"15min":0.29799999999999999},"min":{"1min":0.22700000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":1.361},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.32400000000000001,"15min":0.318},"min":{"1min":0.23200000000000001,"5min":0.216,"15min":0.19},"max":{"1min":0.51100000000000001,"5min":0.51700000000000002,"15min":1.264},"last":0.34499999999999997}]},{"osd":1,"last update":"Wed Apr 24 23:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.32400000000000001,"15min":0.33800000000000002},"min":{"1min":0.26100000000000001,"5min":0.223,"15min":0.20000000000000001},"max":{"1min":0.441,"5min":0.47099999999999997,"15min":6.2469999999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.35199999999999998,"15min":0.34000000000000002},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.22500000000000001},"max":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":1.3859999999999999},"last":0.29999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738810,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313312,"kb_used_data":3668,"kb_used_omap":1,"kb_used_meta":309630,"kb_avail":94058528,"statfs":{"total":96636764160,"available":96315932672,"internally_reserved":0,"allocated":3756032,"data_stored":2013008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317061464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 23:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.29299999999999998,"5min":0.29099999999999998,"15min":0.312},"min":{"1min":0.22,"5min":0.16900000000000001,"15min":0.14399999999999999},"max":{"1min":0.41699999999999998,"5min":0.70699999999999996,"15min":3.0139999999999998},"last":0.246},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.33500000000000002,"15min":0.317},"min":{"1min":0.26400000000000001,"5min":0.23100000000000001,"15min":0.19600000000000001},"max":{"1min":0.50600000000000001,"5min":0.67500000000000004,"15min":0.79100000000000004},"last":0.25700000000000001}]},{"osd":2,"last update":"Wed Apr 24 23:04:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.33900000000000002,"15min":0.33300000000000002},"min":{"1min":0.28100000000000003,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":0.46500000000000002,"5min":0.50700000000000001,"15min":1.415},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.317,"15min":0.32700000000000001},"min":{"1min":0.26500000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47299999999999998,"5min":0.47999999999999998,"15min":1.9159999999999999},"last":0.23300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738811,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301084,"kb_used_data":3728,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94070756,"statfs":{"total":96636764160,"available":96328454144,"internally_reserved":0,"allocated":3817472,"data_stored":2604054,"data_compressed":5763,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 23:03:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.308,"15min":0.29599999999999999},"min":{"1min":0.22600000000000001,"5min":0.17199999999999999,"15min":0.158},"max":{"1min":0.41699999999999998,"5min":0.45700000000000002,"15min":0.46600000000000003},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.34399999999999997,"15min":0.314},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.17000000000000001},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.85199999999999998},"last":0.35099999999999998}]},{"osd":2,"last update":"Wed Apr 24 23:03:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.313,"15min":0.316},"min":{"1min":0.26300000000000001,"5min":0.20799999999999999,"15min":0.185},"max":{"1min":0.46200000000000002,"5min":0.46500000000000002,"15min":0.50700000000000001},"last":0.375},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.31900000000000001,"15min":0.33100000000000002},"min":{"1min":0.25800000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.499,"5min":0.499,"15min":0.54800000000000004},"last":0.315}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5763,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5763,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:04:20.357 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-04-24T22:27:42.276465+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=22, tm_min=27, tm_sec=42, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=4, tm_sec=17, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:04:20.357 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-04-24T22:27:42.276465+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=22, tm_min=27, tm_sec=42, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=4, tm_sec=17, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:04:20.357 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-04-24T22:27:42.276465+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=22, tm_min=27, tm_sec=42, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=4, tm_sec=17, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:04:20.357 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-24T22:27:43.277831+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=22, tm_min=27, tm_sec=43, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=4, tm_sec=17, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:04:20.357 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-04-24T22:27:42.276465+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=22, tm_min=27, tm_sec=42, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=4, tm_sec=17, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:04:20.357 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-24T22:27:42.276465+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=22, tm_min=27, tm_sec=42, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=4, tm_sec=17, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:04:20.357 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-04-24T22:27:42.276465+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=22, tm_min=27, tm_sec=42, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=4, tm_sec=17, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:04:20.358 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-04-24T22:27:42.276465+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=22, tm_min=27, tm_sec=42, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=4, tm_sec=17, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:04:20.358 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-04-24T22:27:42.276465+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=22, tm_min=27, tm_sec=42, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=4, tm_sec=17, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:04:20.358 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-24T23:04:40.358 DEBUG:teuthology.orchestra.run.smithi080:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-24T23:04:40.680 INFO:teuthology.orchestra.run.smithi080.stdout: 2024-04-24T23:04:40.682 INFO:teuthology.orchestra.run.smithi080.stderr:dumped all 2024-04-24T23:04:40.693 INFO:teuthology.orchestra.run.smithi080.stdout:{"pg_ready":true,"pg_map":{"version":1563,"stamp":"2024-04-24T23:04:39.381914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":919560,"kb_used_data":11120,"kb_used_omap":4,"kb_used_meta":908411,"kb_avail":282195960,"statfs":{"total":289910292480,"available":288968663040,"internally_reserved":0,"allocated":11386880,"data_stored":7220438,"data_compressed":11526,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":930212872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002567"},"pg_stats":[{"pgid":"1.5","version":"0'0","reported_seq":466,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-24T23:04:23.917385+0000","last_change":"2024-04-24T23:04:23.917349+0000","last_active":"2024-04-24T23:04:23.917385+0000","last_peered":"2024-04-24T23:04:23.917385+0000","last_clean":"2024-04-24T23:04:23.917385+0000","last_became_active":"2024-04-24T22:27:43.285451+0000","last_became_peered":"2024-04-24T22:27:43.285451+0000","last_unstale":"2024-04-24T23:04:23.917385+0000","last_undegraded":"2024-04-24T23:04:23.917385+0000","last_fullsized":"2024-04-24T23:04:23.917385+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:04:23.917197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:04:23.917197+0000","last_clean_scrub_stamp":"2024-04-24T23:04:23.917197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:27:12.648190+0000","scrub_duration":0.0046647399999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":464,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-24T23:04:27.757932+0000","last_change":"2024-04-24T23:04:27.757903+0000","last_active":"2024-04-24T23:04:27.757932+0000","last_peered":"2024-04-24T23:04:27.757932+0000","last_clean":"2024-04-24T23:04:27.757932+0000","last_became_active":"2024-04-24T22:27:43.285821+0000","last_became_peered":"2024-04-24T22:27:43.285821+0000","last_unstale":"2024-04-24T23:04:27.757932+0000","last_undegraded":"2024-04-24T23:04:27.757932+0000","last_fullsized":"2024-04-24T23:04:27.757932+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:04:27.757788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:04:27.757788+0000","last_clean_scrub_stamp":"2024-04-24T23:04:27.757788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:23.729404+0000","scrub_duration":0.0024421960000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":464,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-24T23:04:21.604327+0000","last_change":"2024-04-24T23:04:21.604297+0000","last_active":"2024-04-24T23:04:21.604327+0000","last_peered":"2024-04-24T23:04:21.604327+0000","last_clean":"2024-04-24T23:04:21.604327+0000","last_became_active":"2024-04-24T22:27:43.284939+0000","last_became_peered":"2024-04-24T22:27:43.284939+0000","last_unstale":"2024-04-24T23:04:21.604327+0000","last_undegraded":"2024-04-24T23:04:21.604327+0000","last_fullsized":"2024-04-24T23:04:21.604327+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:04:21.604185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:04:21.604185+0000","last_clean_scrub_stamp":"2024-04-24T23:04:21.604185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:04:01.641627+0000","scrub_duration":0.001399668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"457'99","reported_seq":594,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-24T23:04:34.848762+0000","last_change":"2024-04-24T23:04:34.848731+0000","last_active":"2024-04-24T23:04:34.848762+0000","last_peered":"2024-04-24T23:04:34.848762+0000","last_clean":"2024-04-24T23:04:34.848762+0000","last_became_active":"2024-04-24T22:27:44.346752+0000","last_became_peered":"2024-04-24T22:27:44.346752+0000","last_unstale":"2024-04-24T23:04:34.848762+0000","last_undegraded":"2024-04-24T23:04:34.848762+0000","last_fullsized":"2024-04-24T23:04:34.848762+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'99","last_scrub_stamp":"2024-04-24T23:04:34.848629+0000","last_deep_scrub":"457'99","last_deep_scrub_stamp":"2024-04-24T23:04:34.848629+0000","last_clean_scrub_stamp":"2024-04-24T23:04:34.848629+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:50:01.704473+0000","scrub_duration":0.011161447,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":464,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-24T23:04:21.970700+0000","last_change":"2024-04-24T23:04:21.970674+0000","last_active":"2024-04-24T23:04:21.970700+0000","last_peered":"2024-04-24T23:04:21.970700+0000","last_clean":"2024-04-24T23:04:21.970700+0000","last_became_active":"2024-04-24T22:27:43.284970+0000","last_became_peered":"2024-04-24T22:27:43.284970+0000","last_unstale":"2024-04-24T23:04:21.970700+0000","last_undegraded":"2024-04-24T23:04:21.970700+0000","last_fullsized":"2024-04-24T23:04:21.970700+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:04:21.970570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:04:21.970570+0000","last_clean_scrub_stamp":"2024-04-24T23:04:21.970570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:43:03.267782+0000","scrub_duration":0.004388561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":464,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-24T23:04:22.604655+0000","last_change":"2024-04-24T23:04:22.604628+0000","last_active":"2024-04-24T23:04:22.604655+0000","last_peered":"2024-04-24T23:04:22.604655+0000","last_clean":"2024-04-24T23:04:22.604655+0000","last_became_active":"2024-04-24T22:27:43.285646+0000","last_became_peered":"2024-04-24T22:27:43.285646+0000","last_unstale":"2024-04-24T23:04:22.604655+0000","last_undegraded":"2024-04-24T23:04:22.604655+0000","last_fullsized":"2024-04-24T23:04:22.604655+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:04:22.604526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:04:22.604526+0000","last_clean_scrub_stamp":"2024-04-24T23:04:22.604526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:03:47.686414+0000","scrub_duration":0.002411875,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":466,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-24T23:04:24.916334+0000","last_change":"2024-04-24T23:04:24.916308+0000","last_active":"2024-04-24T23:04:24.916334+0000","last_peered":"2024-04-24T23:04:24.916334+0000","last_clean":"2024-04-24T23:04:24.916334+0000","last_became_active":"2024-04-24T22:27:43.286197+0000","last_became_peered":"2024-04-24T22:27:43.286197+0000","last_unstale":"2024-04-24T23:04:24.916334+0000","last_undegraded":"2024-04-24T23:04:24.916334+0000","last_fullsized":"2024-04-24T23:04:24.916334+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:04:24.916164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:04:24.916164+0000","last_clean_scrub_stamp":"2024-04-24T23:04:24.916164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:18.134104+0000","scrub_duration":0.0048302880000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":464,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-24T23:04:20.655090+0000","last_change":"2024-04-24T23:04:20.655043+0000","last_active":"2024-04-24T23:04:20.655090+0000","last_peered":"2024-04-24T23:04:20.655090+0000","last_clean":"2024-04-24T23:04:20.655090+0000","last_became_active":"2024-04-24T22:27:43.286126+0000","last_became_peered":"2024-04-24T22:27:43.286126+0000","last_unstale":"2024-04-24T23:04:20.655090+0000","last_undegraded":"2024-04-24T23:04:20.655090+0000","last_fullsized":"2024-04-24T23:04:20.655090+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:04:20.654910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:04:20.654910+0000","last_clean_scrub_stamp":"2024-04-24T23:04:20.654910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:13:59.818908+0000","scrub_duration":0.0030585640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":464,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-04-24T23:04:23.601917+0000","last_change":"2024-04-24T23:04:23.601881+0000","last_active":"2024-04-24T23:04:23.601917+0000","last_peered":"2024-04-24T23:04:23.601917+0000","last_clean":"2024-04-24T23:04:23.601917+0000","last_became_active":"2024-04-24T22:27:43.284828+0000","last_became_peered":"2024-04-24T22:27:43.284828+0000","last_unstale":"2024-04-24T23:04:23.601917+0000","last_undegraded":"2024-04-24T23:04:23.601917+0000","last_fullsized":"2024-04-24T23:04:23.601917+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:04:23.601735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:04:23.601735+0000","last_clean_scrub_stamp":"2024-04-24T23:04:23.601735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:02.928164+0000","scrub_duration":0.0041086669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11526,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738814,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305172,"kb_used_data":3724,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94066668,"statfs":{"total":96636764160,"available":96324268032,"internally_reserved":0,"allocated":3813376,"data_stored":2603376,"data_compressed":5763,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 23:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.29399999999999998,"15min":0.29799999999999999},"min":{"1min":0.22700000000000001,"5min":0.214,"15min":0.18099999999999999},"max":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":1.361},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.32400000000000001,"15min":0.318},"min":{"1min":0.23200000000000001,"5min":0.216,"15min":0.19},"max":{"1min":0.51100000000000001,"5min":0.51700000000000002,"15min":1.264},"last":0.28000000000000003}]},{"osd":1,"last update":"Wed Apr 24 23:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.32400000000000001,"15min":0.33800000000000002},"min":{"1min":0.26100000000000001,"5min":0.223,"15min":0.20000000000000001},"max":{"1min":0.441,"5min":0.47099999999999997,"15min":6.2469999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.35199999999999998,"15min":0.34000000000000002},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.22500000000000001},"max":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":1.3859999999999999},"last":0.32600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738814,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313308,"kb_used_data":3668,"kb_used_omap":1,"kb_used_meta":309630,"kb_avail":94058532,"statfs":{"total":96636764160,"available":96315936768,"internally_reserved":0,"allocated":3756032,"data_stored":2013008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317061464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 23:04:10 2024","interfaces":[{"interface":"back","average":{"1min":0.29299999999999998,"5min":0.29099999999999998,"15min":0.312},"min":{"1min":0.22,"5min":0.16900000000000001,"15min":0.14399999999999999},"max":{"1min":0.41699999999999998,"5min":0.70699999999999996,"15min":3.0139999999999998},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.33500000000000002,"15min":0.317},"min":{"1min":0.26400000000000001,"5min":0.23100000000000001,"15min":0.19600000000000001},"max":{"1min":0.50600000000000001,"5min":0.67500000000000004,"15min":0.79100000000000004},"last":0.34499999999999997}]},{"osd":2,"last update":"Wed Apr 24 23:04:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.33900000000000002,"15min":0.33300000000000002},"min":{"1min":0.28100000000000003,"5min":0.193,"15min":0.17499999999999999},"max":{"1min":0.46500000000000002,"5min":0.50700000000000001,"15min":1.415},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.317,"15min":0.32700000000000001},"min":{"1min":0.26500000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47299999999999998,"5min":0.47999999999999998,"15min":1.9159999999999999},"last":0.314}]}]},{"osd":1,"up_from":8,"seq":34359738815,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301080,"kb_used_data":3728,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94070760,"statfs":{"total":96636764160,"available":96328458240,"internally_reserved":0,"allocated":3817472,"data_stored":2604054,"data_compressed":5763,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 23:03:55 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.308,"15min":0.29599999999999999},"min":{"1min":0.22600000000000001,"5min":0.17199999999999999,"15min":0.158},"max":{"1min":0.41699999999999998,"5min":0.45700000000000002,"15min":0.46600000000000003},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.34399999999999997,"15min":0.314},"min":{"1min":0.24399999999999999,"5min":0.218,"15min":0.17000000000000001},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.85199999999999998},"last":0.49399999999999999}]},{"osd":2,"last update":"Wed Apr 24 23:03:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.313,"15min":0.316},"min":{"1min":0.26300000000000001,"5min":0.20799999999999999,"15min":0.185},"max":{"1min":0.46200000000000002,"5min":0.46500000000000002,"15min":0.50700000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.31900000000000001,"15min":0.33100000000000002},"min":{"1min":0.25800000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.499,"5min":0.499,"15min":0.54800000000000004},"last":0.439}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5763,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5763,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:04:40.695 DEBUG:teuthology.orchestra.run.smithi080:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-04-24T23:04:41.022 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-24T23:04:41.022 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-24T23:04:41.022 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-04-24T23:04:41.022 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:04:41.062 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-04-24T23:04:41.062 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-04-24T23:04:41.062 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:04:41.135 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-04-24T23:04:41.135 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-04-24T23:04:41.135 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:04:41.209 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-04-24T23:04:41.209 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-24T23:04:41.210 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-04-24T23:04:41.210 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:04:41.274 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-04-24T23:04:41.274 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-24T23:04:41.274 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-04-24T23:04:41.274 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:04:41.332 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-04-24T23:04:41.332 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-24T23:04:41.332 DEBUG:teuthology.orchestra.run.smithi080:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-04-24T23:04:41.407 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi080.front.sepia.ceph.com 2024-04-24T23:04:41.407 DEBUG:teuthology.orchestra.run.smithi080:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-04-24T23:04:41.582 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi080.front.sepia.ceph.com 2024-04-24T23:04:41.582 DEBUG:teuthology.orchestra.run.smithi080:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-04-24T23:04:41.683 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi080.front.sepia.ceph.com 2024-04-24T23:04:41.683 DEBUG:teuthology.orchestra.run.smithi080:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-04-24T23:04:41.770 INFO:tasks.ceph:Archiving mon data... 2024-04-24T23:04:41.770 DEBUG:teuthology.misc:Transferring archived files from smithi080:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/yuriw-2024-04-24_21:25:53-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7672255/data/cluster1.mon.a.tgz 2024-04-24T23:04:41.770 DEBUG:teuthology.orchestra.run.smithi080:> mktemp 2024-04-24T23:04:41.794 INFO:teuthology.orchestra.run.smithi080.stdout:/tmp/tmp.6c5amdTs5w 2024-04-24T23:04:41.794 DEBUG:teuthology.orchestra.run.smithi080:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.6c5amdTs5w 2024-04-24T23:04:41.962 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 0666 /tmp/tmp.6c5amdTs5w 2024-04-24T23:04:42.080 DEBUG:teuthology.orchestra.remote:smithi080:/tmp/tmp.6c5amdTs5w is 378KB 2024-04-24T23:04:42.131 DEBUG:teuthology.orchestra.run.smithi080:> rm -fr /tmp/tmp.6c5amdTs5w 2024-04-24T23:04:42.151 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-24T23:04:42.151 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T23:04:42.200 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T23:04:42.237 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-04-24T23:04:42.237 DEBUG:teuthology.orchestra.run.smithi080:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-24T23:04:42.275 DEBUG:teuthology.orchestra.run.smithi134:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-24T23:04:42.300 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-24T23:04:42.314 INFO:teuthology.orchestra.run.smithi134.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-24T23:04:42.320 INFO:tasks.ceph:Archiving crash dumps... 2024-04-24T23:04:42.328 DEBUG:teuthology.misc:Transferring archived files from smithi080:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-24_21:25:53-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7672255/remote/smithi080/crash 2024-04-24T23:04:42.331 DEBUG:teuthology.orchestra.run.smithi080:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-24T23:04:42.399 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-24_21:25:53-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7672255/remote/smithi134/crash 2024-04-24T23:04:42.427 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-24T23:04:42.463 INFO:tasks.ceph:Compressing logs... 2024-04-24T23:04:42.463 DEBUG:teuthology.orchestra.run.smithi080:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T23:04:42.474 DEBUG:teuthology.orchestra.run.smithi134:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T23:04:42.499 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-04-24T23:04:42.506 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46340.log 2024-04-24T23:04:42.507 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2024-04-24T23:04:42.517 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.46340.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46340.log.gz 2024-04-24T23:04:42.517 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46238.log 2024-04-24T23:04:42.517 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48360.log 2024-04-24T23:04:42.517 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster2-mgr.x.log: /var/log/ceph/cluster1-client.admin.46238.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48213.log 2024-04-24T23:04:42.517 INFO:teuthology.orchestra.run.smithi080.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46238.log.gz 2024-04-24T23:04:42.517 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.48360.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48262.log 2024-04-24T23:04:42.517 INFO:teuthology.orchestra.run.smithi080.stderr: 28.5% -- replaced with /var/log/ceph/cluster1-client.admin.48360.log.gz 2024-04-24T23:04:42.517 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.48213.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/cluster1-client.admin.48213.log.gz-- 2024-04-24T23:04:42.517 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.48311.log 2024-04-24T23:04:42.517 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.48262.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-04-24T23:04:42.517 INFO:teuthology.orchestra.run.smithi080.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48262.log.gz 2024-04-24T23:04:42.518 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.48311.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42259.log 2024-04-24T23:04:42.518 INFO:teuthology.orchestra.run.smithi080.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48311.log.gz 2024-04-24T23:04:42.518 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48507.log 2024-04-24T23:04:42.518 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.42259.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.42259.log.gz 2024-04-24T23:04:42.518 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46624.log 2024-04-24T23:04:42.518 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46089.log 2024-04-24T23:04:42.518 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.48507.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48507.log.gz 2024-04-24T23:04:42.518 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.46624.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46679.log 2024-04-24T23:04:42.518 INFO:teuthology.orchestra.run.smithi080.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46624.log.gz 2024-04-24T23:04:42.518 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.46089.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46089.log.gz 2024-04-24T23:04:42.518 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46189.log 2024-04-24T23:04:42.518 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46032.log 2024-04-24T23:04:42.518 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.46679.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46679.log.gz 2024-04-24T23:04:42.518 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.46189.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42672.log 2024-04-24T23:04:42.519 INFO:teuthology.orchestra.run.smithi080.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46189.log.gz 2024-04-24T23:04:42.519 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.46032.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46032.log.gz 2024-04-24T23:04:42.519 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48458.log 2024-04-24T23:04:42.519 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-04-24T23:04:42.519 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.42672.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.42672.log.gz 2024-04-24T23:04:42.519 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.48458.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.37442.log 2024-04-24T23:04:42.519 INFO:teuthology.orchestra.run.smithi080.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48458.log.gz 2024-04-24T23:04:42.519 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48654.log 2024-04-24T23:04:42.519 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/ceph.tmp-client.admin.37442.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.37442.log.gz 2024-04-24T23:04:42.519 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.48654.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48654.log.gz 2024-04-24T23:04:42.519 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-04-24T23:04:42.519 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48801.log 2024-04-24T23:04:42.519 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-osd.2.log: /var/log/ceph/cluster1-client.admin.48801.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48801.log.gz 2024-04-24T23:04:42.519 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48703.log 2024-04-24T23:04:42.519 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48556.log 2024-04-24T23:04:42.519 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.48703.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48703.log.gz 2024-04-24T23:04:42.520 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.48556.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48556.log.gz 2024-04-24T23:04:42.520 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46820.log 2024-04-24T23:04:42.520 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46449.log 2024-04-24T23:04:42.520 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.46820.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46820.log.gz 2024-04-24T23:04:42.520 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.46449.log: 86.9% -- replaced with /var/log/ceph/cluster1.log.gz 2024-04-24T23:04:42.520 INFO:teuthology.orchestra.run.smithi080.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46449.log.gz 2024-04-24T23:04:42.520 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48752.log 2024-04-24T23:04:42.520 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46650.log 2024-04-24T23:04:42.520 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.48752.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46339.log 2024-04-24T23:04:42.520 INFO:teuthology.orchestra.run.smithi080.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48752.log.gz 2024-04-24T23:04:42.520 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.46650.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46650.log.gz 2024-04-24T23:04:42.520 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.46339.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46339.log.gz 2024-04-24T23:04:42.520 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46338.log 2024-04-24T23:04:42.520 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42443.log 2024-04-24T23:04:42.521 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.46338.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46338.log.gz 2024-04-24T23:04:42.521 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.42443.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.42443.log.gz 2024-04-24T23:04:42.521 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-04-24T23:04:42.521 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42499.log 2024-04-24T23:04:42.521 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-04-24T23:04:42.521 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.42499.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.42499.log.gz 2024-04-24T23:04:42.521 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1.audit.log: 92.4% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-04-24T23:04:42.521 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45963.log 2024-04-24T23:04:42.521 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46535.log 2024-04-24T23:04:42.522 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.45963.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.45963.log.gz 2024-04-24T23:04:42.523 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.46535.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46535.log.gz 2024-04-24T23:04:42.523 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46480.log 2024-04-24T23:04:42.523 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42747.log 2024-04-24T23:04:42.524 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.46480.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46480.log.gz 2024-04-24T23:04:42.524 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.42747.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.42747.log.gz 2024-04-24T23:04:42.524 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-04-24T23:04:42.524 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48848.log 2024-04-24T23:04:42.524 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-osd.1.log: /var/log/ceph/cluster1-client.admin.48848.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48848.log.gz 2024-04-24T23:04:42.524 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48409.log 2024-04-24T23:04:42.524 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.48409.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48409.log.gz 2024-04-24T23:04:42.524 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46140.log 2024-04-24T23:04:42.524 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.46140.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46140.log.gz 2024-04-24T23:04:42.524 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48605.log 2024-04-24T23:04:42.524 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.48605.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48605.log.gz 2024-04-24T23:04:42.524 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46771.log 2024-04-24T23:04:42.524 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.46771.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46771.log.gz 2024-04-24T23:04:42.525 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42547.log 2024-04-24T23:04:42.525 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.42547.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.42547.log.gz 2024-04-24T23:04:42.525 INFO:teuthology.orchestra.run.smithi080.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46869.log 2024-04-24T23:04:42.525 INFO:teuthology.orchestra.run.smithi080.stderr:/var/log/ceph/cluster1-client.admin.46869.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46869.log.gz 2024-04-24T23:04:42.546 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85458.log 2024-04-24T23:04:42.547 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54607.log 2024-04-24T23:04:42.547 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.85458.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52062.log 2024-04-24T23:04:42.547 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85458.log.gz 2024-04-24T23:04:42.547 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.54607.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50205.log 2024-04-24T23:04:42.547 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75483.log 2024-04-24T23:04:42.547 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.52062.log: /var/log/ceph/cluster1-client.mirror.50205.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.56105.log 2024-04-24T23:04:42.547 INFO:teuthology.orchestra.run.smithi134.stderr: 81.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50205.log.gz 2024-04-24T23:04:42.548 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75483.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70462.log 2024-04-24T23:04:42.548 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.2.56105.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84371.log 2024-04-24T23:04:42.548 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.70462.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65579.log 2024-04-24T23:04:42.548 INFO:teuthology.orchestra.run.smithi134.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70462.log.gz 2024-04-24T23:04:42.548 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84371.log: /var/log/ceph/cluster2-client.mirror.65579.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84371.log.gz 2024-04-24T23:04:42.549 INFO:teuthology.orchestra.run.smithi134.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65579.log.gz 2024-04-24T23:04:42.549 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.58517.log 2024-04-24T23:04:42.549 INFO:teuthology.orchestra.run.smithi134.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75483.log.gz 2024-04-24T23:04:42.550 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82195.log 2024-04-24T23:04:42.550 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.0.58517.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74011.log 2024-04-24T23:04:42.550 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.82195.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59118.log 2024-04-24T23:04:42.551 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.82195.log.gz 2024-04-24T23:04:42.552 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.74011.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77390.log 2024-04-24T23:04:42.552 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59118.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74011.log.gz 2024-04-24T23:04:42.552 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54607.log.gz 2024-04-24T23:04:42.552 INFO:teuthology.orchestra.run.smithi134.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59118.log.gz 2024-04-24T23:04:42.552 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77390.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52062.log.gz 2024-04-24T23:04:42.552 INFO:teuthology.orchestra.run.smithi134.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.0.58517.log.gz 2024-04-24T23:04:42.552 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74362.log 2024-04-24T23:04:42.552 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.77390.log.gz 2024-04-24T23:04:42.552 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55164.log 2024-04-24T23:04:42.554 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.74362.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60880.log 2024-04-24T23:04:42.554 INFO:teuthology.orchestra.run.smithi134.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.2.56105.log.gz 2024-04-24T23:04:42.554 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.55164.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65799.log 2024-04-24T23:04:42.555 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60880.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73443.log 2024-04-24T23:04:42.555 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65799.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80777.log 2024-04-24T23:04:42.555 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73443.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62467.log 2024-04-24T23:04:42.555 INFO:teuthology.orchestra.run.smithi134.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74362.log.gz 2024-04-24T23:04:42.555 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80777.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.80777.log.gz 2024-04-24T23:04:42.555 INFO:teuthology.orchestra.run.smithi134.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73443.log.gz 2024-04-24T23:04:42.555 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62467.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65799.log.gz 2024-04-24T23:04:42.556 INFO:teuthology.orchestra.run.smithi134.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62467.log.gz 2024-04-24T23:04:42.556 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68809.log 2024-04-24T23:04:42.556 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68129.log 2024-04-24T23:04:42.557 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68809.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64690.log 2024-04-24T23:04:42.557 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68129.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56973.log 2024-04-24T23:04:42.557 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.64690.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49146.log 2024-04-24T23:04:42.558 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56973.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68129.log.gz 2024-04-24T23:04:42.558 INFO:teuthology.orchestra.run.smithi134.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64690.log.gz 2024-04-24T23:04:42.558 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49146.log: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49146.log.gz 2024-04-24T23:04:42.558 INFO:teuthology.orchestra.run.smithi134.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68809.log.gz 2024-04-24T23:04:42.558 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78316.log 2024-04-24T23:04:42.558 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55164.log.gz 2024-04-24T23:04:42.558 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57115.log 2024-04-24T23:04:42.558 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78316.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68375.log 2024-04-24T23:04:42.559 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57115.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65182.log 2024-04-24T23:04:42.559 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68375.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51093.log 2024-04-24T23:04:42.560 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65182.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57115.log.gz 2024-04-24T23:04:42.560 INFO:teuthology.orchestra.run.smithi134.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65182.log.gz 2024-04-24T23:04:42.560 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51093.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68375.log.gz 2024-04-24T23:04:42.560 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65944.log 2024-04-24T23:04:42.560 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81922.log 2024-04-24T23:04:42.561 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.65944.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.78316.log.gz 2024-04-24T23:04:42.561 INFO:teuthology.orchestra.run.smithi134.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51093.log.gz 2024-04-24T23:04:42.561 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81922.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65944.log.gz 2024-04-24T23:04:42.561 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81181.log 2024-04-24T23:04:42.561 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65230.log 2024-04-24T23:04:42.561 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81181.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68692.log 2024-04-24T23:04:42.562 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.65230.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65658.log 2024-04-24T23:04:42.562 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68692.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.81181.log.gz 2024-04-24T23:04:42.562 INFO:teuthology.orchestra.run.smithi134.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65230.log.gz 2024-04-24T23:04:42.562 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65658.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68692.log.gz 2024-04-24T23:04:42.562 INFO:teuthology.orchestra.run.smithi134.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65658.log.gz 2024-04-24T23:04:42.562 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65319.log 2024-04-24T23:04:42.562 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75649.log 2024-04-24T23:04:42.562 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.65319.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78868.log 2024-04-24T23:04:42.563 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75649.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52571.log 2024-04-24T23:04:42.563 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.78868.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75649.log.gz 2024-04-24T23:04:42.565 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.78868.log.gz 2024-04-24T23:04:42.565 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.52571.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52571.log.gz 2024-04-24T23:04:42.565 INFO:teuthology.orchestra.run.smithi134.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65319.log.gz 2024-04-24T23:04:42.565 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52794.log 2024-04-24T23:04:42.566 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66666.log 2024-04-24T23:04:42.566 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.52794.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53018.log 2024-04-24T23:04:42.566 INFO:teuthology.orchestra.run.smithi134.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.52794.log.gz 2024-04-24T23:04:42.566 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.66666.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71595.log 2024-04-24T23:04:42.566 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55101.log 2024-04-24T23:04:42.566 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.53018.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66666.log.gz 2024-04-24T23:04:42.566 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71595.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53018.log.gz 2024-04-24T23:04:42.567 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55101.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71595.log.gz 2024-04-24T23:04:42.567 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78508.log 2024-04-24T23:04:42.567 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73701.log 2024-04-24T23:04:42.569 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.78508.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85752.log 2024-04-24T23:04:42.569 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78508.log.gz 2024-04-24T23:04:42.569 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73701.log: /var/log/ceph/cluster2-client.admin.85752.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73701.log.gz 2024-04-24T23:04:42.569 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85752.log.gz 2024-04-24T23:04:42.569 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61868.log 2024-04-24T23:04:42.569 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56623.log 2024-04-24T23:04:42.571 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61868.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76967.log 2024-04-24T23:04:42.571 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56623.log: /var/log/ceph/cluster2-client.mirror.76967.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56623.log.gz 2024-04-24T23:04:42.571 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55101.log.gz 2024-04-24T23:04:42.571 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54650.log 2024-04-24T23:04:42.571 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72622.log 2024-04-24T23:04:42.577 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.54650.log: /var/log/ceph/cluster2-client.mirror.72622.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76967.log.gz 2024-04-24T23:04:42.577 INFO:teuthology.orchestra.run.smithi134.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72622.log.gz 2024-04-24T23:04:42.577 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64123.log 2024-04-24T23:04:42.577 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58968.log 2024-04-24T23:04:42.577 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64123.log: /var/log/ceph/cluster2-client.mirror.58968.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58968.log.gz 2024-04-24T23:04:42.577 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77750.log 2024-04-24T23:04:42.577 INFO:teuthology.orchestra.run.smithi134.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64123.log.gz 2024-04-24T23:04:42.577 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77750.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.77750.log.gz 2024-04-24T23:04:42.577 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75153.log 2024-04-24T23:04:42.577 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60508.log 2024-04-24T23:04:42.578 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75153.log: /var/log/ceph/cluster2-client.mirror.60508.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75153.log.gz 2024-04-24T23:04:42.578 INFO:teuthology.orchestra.run.smithi134.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60508.log.gz 2024-04-24T23:04:42.578 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56643.log 2024-04-24T23:04:42.578 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54650.log.gz 2024-04-24T23:04:42.579 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63118.log 2024-04-24T23:04:42.579 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56643.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77009.log 2024-04-24T23:04:42.579 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.63118.log: /var/log/ceph/cluster2-client.mirror.77009.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56643.log.gz 2024-04-24T23:04:42.579 INFO:teuthology.orchestra.run.smithi134.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63118.log.gz 2024-04-24T23:04:42.579 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75215.log 2024-04-24T23:04:42.579 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.77009.log.gz 2024-04-24T23:04:42.579 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72749.log 2024-04-24T23:04:42.580 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75215.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75897.log 2024-04-24T23:04:42.580 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.72749.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75215.log.gz 2024-04-24T23:04:42.580 INFO:teuthology.orchestra.run.smithi134.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.72749.log.gz 2024-04-24T23:04:42.580 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75897.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77791.log 2024-04-24T23:04:42.581 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75897.log.gz 2024-04-24T23:04:42.581 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72511.log 2024-04-24T23:04:42.581 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77791.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56280.log 2024-04-24T23:04:42.581 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.72511.log: /var/log/ceph/cluster1-client.mirror.56280.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77791.log.gz 2024-04-24T23:04:42.581 INFO:teuthology.orchestra.run.smithi134.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.72511.log.gz 2024-04-24T23:04:42.584 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69668.log 2024-04-24T23:04:42.584 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79701.log 2024-04-24T23:04:42.585 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69668.log: /var/log/ceph/cluster1-client.mirror.79701.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69668.log.gz 2024-04-24T23:04:42.585 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.79701.log.gz 2024-04-24T23:04:42.585 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73511.log 2024-04-24T23:04:42.586 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68851.log 2024-04-24T23:04:42.586 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.73511.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73511.log.gz 2024-04-24T23:04:42.586 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68851.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56280.log.gz 2024-04-24T23:04:42.587 INFO:teuthology.orchestra.run.smithi134.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68851.log.gz 2024-04-24T23:04:42.587 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67596.log 2024-04-24T23:04:42.587 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80376.log 2024-04-24T23:04:42.588 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67596.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58007.log 2024-04-24T23:04:42.588 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80376.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67596.log.gz 2024-04-24T23:04:42.588 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.80376.log.gz 2024-04-24T23:04:42.598 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58007.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75835.log 2024-04-24T23:04:42.598 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60900.log 2024-04-24T23:04:42.599 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75835.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75835.log.gz 2024-04-24T23:04:42.599 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60900.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60900.log.gz 2024-04-24T23:04:42.599 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.58058.log 2024-04-24T23:04:42.600 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55080.log 2024-04-24T23:04:42.604 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.2.58058.log: /var/log/ceph/cluster1-client.mirror.55080.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55080.log.gz 2024-04-24T23:04:42.604 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51470.log 2024-04-24T23:04:42.604 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51470.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51470.log.gz 2024-04-24T23:04:42.604 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61242.log 2024-04-24T23:04:42.604 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61242.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61242.log.gz 2024-04-24T23:04:42.604 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51586.log 2024-04-24T23:04:42.605 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.51586.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51586.log.gz 2024-04-24T23:04:42.605 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84298.log 2024-04-24T23:04:42.605 INFO:teuthology.orchestra.run.smithi134.stderr: 93.7% -- replaced with /var/log/ceph/cluster1-client.mirror.2.58058.log.gz 2024-04-24T23:04:42.605 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57806.log 2024-04-24T23:04:42.607 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84298.log: /var/log/ceph/cluster2-client.mirror.57806.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57806.log.gz 2024-04-24T23:04:42.607 INFO:teuthology.orchestra.run.smithi134.stderr:gzip 91.4%-5 --verbose -- replaced with /var/log/ceph/cluster2-client.mirror.84298.log.gz-- 2024-04-24T23:04:42.607 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78151.log 2024-04-24T23:04:42.608 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78151.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58473.log 2024-04-24T23:04:42.608 INFO:teuthology.orchestra.run.smithi134.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.78151.log.gz 2024-04-24T23:04:42.611 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58473.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79341.log 2024-04-24T23:04:42.612 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79341.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79341.log.gz 2024-04-24T23:04:42.612 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52387.log 2024-04-24T23:04:42.612 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.52387.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52387.log.gz 2024-04-24T23:04:42.612 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68954.log 2024-04-24T23:04:42.615 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.68954.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68954.log.gz 2024-04-24T23:04:42.615 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61084.log 2024-04-24T23:04:42.616 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61084.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61084.log.gz 2024-04-24T23:04:42.616 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83977.log 2024-04-24T23:04:42.616 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83977.log: 77.7% -- replaced with /var/log/ceph/cluster2-client.mirror.83977.log.gz 2024-04-24T23:04:42.616 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79318.log 2024-04-24T23:04:42.617 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79318.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79318.log.gz 2024-04-24T23:04:42.617 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73821.log 2024-04-24T23:04:42.618 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.73821.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.73821.log.gz 2024-04-24T23:04:42.618 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69026.log 2024-04-24T23:04:42.619 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.69026.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69026.log.gz 2024-04-24T23:04:42.619 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80713.log 2024-04-24T23:04:42.620 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80713.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.80713.log.gz 2024-04-24T23:04:42.620 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79971.log 2024-04-24T23:04:42.620 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79971.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.79971.log.gz 2024-04-24T23:04:42.621 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43083.log 2024-04-24T23:04:42.621 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.43083.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.43083.log.gz 2024-04-24T23:04:42.621 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50480.log 2024-04-24T23:04:42.622 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50480.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50480.log.gz 2024-04-24T23:04:42.622 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83633.log 2024-04-24T23:04:42.623 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.83633.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.83633.log.gz 2024-04-24T23:04:42.623 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.60215.log 2024-04-24T23:04:42.628 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.0.60215.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.60215.log.gz 2024-04-24T23:04:42.628 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61262.log 2024-04-24T23:04:42.629 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61262.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59047.log 2024-04-24T23:04:42.629 INFO:teuthology.orchestra.run.smithi134.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61262.log.gz 2024-04-24T23:04:42.629 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59047.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83811.log 2024-04-24T23:04:42.630 INFO:teuthology.orchestra.run.smithi134.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59047.log.gz 2024-04-24T23:04:42.632 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83811.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.83811.log.gz 2024-04-24T23:04:42.632 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64224.log 2024-04-24T23:04:42.633 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64224.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64224.log.gz 2024-04-24T23:04:42.633 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78913.log 2024-04-24T23:04:42.634 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.78913.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.78913.log.gz 2024-04-24T23:04:42.634 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85178.log 2024-04-24T23:04:42.635 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.85178.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85178.log.gz 2024-04-24T23:04:42.635 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48768.log 2024-04-24T23:04:42.635 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.48768.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.48768.log.gz 2024-04-24T23:04:42.635 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55761.log 2024-04-24T23:04:42.641 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55761.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55761.log.gz 2024-04-24T23:04:42.641 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82239.log 2024-04-24T23:04:42.642 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82239.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82239.log.gz 2024-04-24T23:04:42.642 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72096.log 2024-04-24T23:04:42.645 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72096.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72096.log.gz 2024-04-24T23:04:42.645 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84132.log 2024-04-24T23:04:42.646 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84132.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84132.log.gz 2024-04-24T23:04:42.646 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75545.log 2024-04-24T23:04:42.648 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75545.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75545.log.gz 2024-04-24T23:04:42.648 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66238.log 2024-04-24T23:04:42.649 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77493.log 2024-04-24T23:04:42.649 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.66238.log: /var/log/ceph/cluster2-client.mirror.77493.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.66238.log.gz 2024-04-24T23:04:42.650 INFO:teuthology.orchestra.run.smithi134.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77493.log.gz 2024-04-24T23:04:42.650 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59660.log 2024-04-24T23:04:42.651 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59660.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59660.log.gz 2024-04-24T23:04:42.651 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58517.log 2024-04-24T23:04:42.652 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.58517.log: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58517.log.gz 2024-04-24T23:04:42.652 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46563.log 2024-04-24T23:04:42.653 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.46563.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.46563.log.gz 2024-04-24T23:04:42.653 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70414.log 2024-04-24T23:04:42.653 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.70414.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70414.log.gz 2024-04-24T23:04:42.654 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84392.log 2024-04-24T23:04:42.655 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84392.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84392.log.gz 2024-04-24T23:04:42.655 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57789.log 2024-04-24T23:04:42.655 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57789.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57789.log.gz 2024-04-24T23:04:42.656 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48573.log 2024-04-24T23:04:42.656 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.48573.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48573.log.gz 2024-04-24T23:04:42.656 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80106.log 2024-04-24T23:04:42.657 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80106.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80106.log.gz 2024-04-24T23:04:42.657 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82735.log 2024-04-24T23:04:42.658 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82735.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82735.log.gz 2024-04-24T23:04:42.658 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63159.log 2024-04-24T23:04:42.660 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.63159.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63159.log.gz 2024-04-24T23:04:42.660 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52336.log 2024-04-24T23:04:42.663 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52336.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52336.log.gz 2024-04-24T23:04:42.663 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83052.log 2024-04-24T23:04:42.664 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83052.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83052.log.gz 2024-04-24T23:04:42.664 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69930.log 2024-04-24T23:04:42.664 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.69930.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69930.log.gz 2024-04-24T23:04:42.665 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76430.log 2024-04-24T23:04:42.666 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76430.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76430.log.gz 2024-04-24T23:04:42.666 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65920.log 2024-04-24T23:04:42.667 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65920.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65920.log.gz 2024-04-24T23:04:42.667 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51028.log 2024-04-24T23:04:42.673 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.51028.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51028.log.gz 2024-04-24T23:04:42.673 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79363.log 2024-04-24T23:04:42.674 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79363.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79363.log.gz 2024-04-24T23:04:42.674 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78172.log 2024-04-24T23:04:42.675 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78172.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.78172.log.gz 2024-04-24T23:04:42.675 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68264.log 2024-04-24T23:04:42.675 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51949.log 2024-04-24T23:04:42.676 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.68264.log: /var/log/ceph/cluster1-client.mirror.51949.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.68264.log.gz 2024-04-24T23:04:42.676 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.51949.log.gz 2024-04-24T23:04:42.676 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57337.log 2024-04-24T23:04:42.677 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83711.log 2024-04-24T23:04:42.677 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57337.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57337.log.gz 2024-04-24T23:04:42.678 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.83711.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.83711.log.gz 2024-04-24T23:04:42.678 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61911.log 2024-04-24T23:04:42.678 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.61911.log: 85.2% -- replaced with /var/log/ceph/cluster1-client.mirror.61911.log.gz 2024-04-24T23:04:42.678 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78688.log 2024-04-24T23:04:42.679 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69430.log 2024-04-24T23:04:42.680 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.78688.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78688.log.gz 2024-04-24T23:04:42.680 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.69430.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69430.log.gz 2024-04-24T23:04:42.680 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65706.log 2024-04-24T23:04:42.681 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.65706.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65706.log.gz 2024-04-24T23:04:42.681 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.48090.log 2024-04-24T23:04:42.681 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48090.log 2024-04-24T23:04:42.695 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.3.48090.log: /var/log/ceph/cluster2-client.mirror.48090.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60049.log 2024-04-24T23:04:42.702 INFO:teuthology.orchestra.run.smithi080.stderr: 91.0% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-04-24T23:04:42.709 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60049.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60049.log.gz 2024-04-24T23:04:42.720 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84152.log 2024-04-24T23:04:42.731 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84152.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48649.log 2024-04-24T23:04:42.732 INFO:teuthology.orchestra.run.smithi134.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84152.log.gz 2024-04-24T23:04:42.744 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.48649.log: 54.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48649.log.gz 2024-04-24T23:04:42.744 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55277.log 2024-04-24T23:04:42.756 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.55277.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55277.log.gzgzip 2024-04-24T23:04:42.756 INFO:teuthology.orchestra.run.smithi134.stderr:-5 --verbose -- /var/log/ceph/cluster1-client.mirror.80196.log 2024-04-24T23:04:42.768 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80196.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.80196.log.gz 2024-04-24T23:04:42.768 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73321.log 2024-04-24T23:04:42.769 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60880.log.gz 2024-04-24T23:04:42.770 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.73321.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.73321.log.gz 2024-04-24T23:04:42.770 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66952.log 2024-04-24T23:04:42.771 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.66952.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66952.log.gz 2024-04-24T23:04:42.771 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67430.log 2024-04-24T23:04:42.772 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.67430.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.67430.log.gz 2024-04-24T23:04:42.772 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79431.log 2024-04-24T23:04:42.773 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79431.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.79431.log.gz 2024-04-24T23:04:42.773 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55008.log 2024-04-24T23:04:42.774 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.55008.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55008.log.gz 2024-04-24T23:04:42.774 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53591.log 2024-04-24T23:04:42.775 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.53591.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53591.log.gz 2024-04-24T23:04:42.775 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68018.log 2024-04-24T23:04:42.777 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.68018.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.68018.log.gz 2024-04-24T23:04:42.777 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65730.log 2024-04-24T23:04:42.777 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.65730.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65730.log.gz 2024-04-24T23:04:42.777 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.77660.log 2024-04-24T23:04:42.788 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.0.77660.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52149.log 2024-04-24T23:04:42.796 INFO:teuthology.orchestra.run.smithi080.stderr: 91.0% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-04-24T23:04:42.810 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52149.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.77660.log.gz 2024-04-24T23:04:42.810 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58951.log 2024-04-24T23:04:42.811 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58951.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58951.log.gz 2024-04-24T23:04:42.811 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59719.log 2024-04-24T23:04:42.822 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59719.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85153.log 2024-04-24T23:04:42.832 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58473.log.gz 2024-04-24T23:04:42.842 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.85153.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85153.log.gz 2024-04-24T23:04:42.842 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63046.log 2024-04-24T23:04:42.853 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.63046.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63046.log.gz 2024-04-24T23:04:42.853 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65034.log 2024-04-24T23:04:42.883 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56973.log.gz 2024-04-24T23:04:42.884 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65034.log: 73.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65034.log.gz 2024-04-24T23:04:42.884 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77514.log 2024-04-24T23:04:42.884 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48090.log.gz 2024-04-24T23:04:42.885 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77514.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.77514.log.gz 2024-04-24T23:04:42.885 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66452.log 2024-04-24T23:04:42.886 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.66452.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66452.log.gz 2024-04-24T23:04:42.886 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77091.log 2024-04-24T23:04:42.889 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77091.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77091.log.gz 2024-04-24T23:04:42.889 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79183.log 2024-04-24T23:04:42.889 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79183.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.79183.log.gz 2024-04-24T23:04:42.889 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75855.log 2024-04-24T23:04:42.892 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75855.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75855.log.gz 2024-04-24T23:04:42.892 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79611.log 2024-04-24T23:04:42.893 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79611.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79611.log.gz 2024-04-24T23:04:42.893 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55695.log 2024-04-24T23:04:42.899 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.55695.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55695.log.gz 2024-04-24T23:04:42.899 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66404.log 2024-04-24T23:04:42.900 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66404.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66404.log.gz 2024-04-24T23:04:42.900 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71740.log 2024-04-24T23:04:42.901 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.71740.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71740.log.gz 2024-04-24T23:04:42.901 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59294.log 2024-04-24T23:04:42.902 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59294.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59294.log.gz 2024-04-24T23:04:42.902 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55781.log 2024-04-24T23:04:42.908 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.55781.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55781.log.gz 2024-04-24T23:04:42.908 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68066.log 2024-04-24T23:04:42.909 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.68066.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.68066.log.gz 2024-04-24T23:04:42.909 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66759.log 2024-04-24T23:04:42.911 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66759.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66759.log.gz 2024-04-24T23:04:42.912 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52021.log 2024-04-24T23:04:42.913 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.52021.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52021.log.gz 2024-04-24T23:04:42.913 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55186.log 2024-04-24T23:04:42.914 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55186.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55186.log.gz 2024-04-24T23:04:42.914 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.83208.log 2024-04-24T23:04:42.918 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.3.83208.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.3.83208.log.gz 2024-04-24T23:04:42.918 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80832.log 2024-04-24T23:04:42.919 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80832.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.80832.log.gz 2024-04-24T23:04:42.919 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62901.log 2024-04-24T23:04:42.921 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62901.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62901.log.gz 2024-04-24T23:04:42.921 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47223.log 2024-04-24T23:04:42.922 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.47223.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.47223.log.gz 2024-04-24T23:04:42.922 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43528.log 2024-04-24T23:04:42.923 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.43528.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.43528.log.gz 2024-04-24T23:04:42.923 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43241.log 2024-04-24T23:04:42.923 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.43241.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.43241.log.gz 2024-04-24T23:04:42.923 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62663.log 2024-04-24T23:04:42.926 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62663.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62663.log.gz 2024-04-24T23:04:42.926 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74124.log 2024-04-24T23:04:42.928 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.74124.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.74124.log.gz 2024-04-24T23:04:42.928 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76471.log 2024-04-24T23:04:42.931 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76471.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76471.log.gz 2024-04-24T23:04:42.931 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63002.log 2024-04-24T23:04:42.933 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63002.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63002.log.gz 2024-04-24T23:04:42.933 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71833.log 2024-04-24T23:04:42.936 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71833.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71833.log.gz 2024-04-24T23:04:42.936 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74225.log 2024-04-24T23:04:42.937 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74225.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74225.log.gz 2024-04-24T23:04:42.937 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79476.log 2024-04-24T23:04:42.938 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79476.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79476.log.gz 2024-04-24T23:04:42.938 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84904.log 2024-04-24T23:04:42.941 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84904.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.84904.log.gz 2024-04-24T23:04:42.941 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67144.log 2024-04-24T23:04:42.942 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.67144.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.67144.log.gz 2024-04-24T23:04:42.943 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55738.log 2024-04-24T23:04:42.948 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.55738.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55738.log.gz 2024-04-24T23:04:42.948 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57646.log 2024-04-24T23:04:42.950 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57646.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57646.log.gz 2024-04-24T23:04:42.950 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.61911.log 2024-04-24T23:04:42.956 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.2.61911.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.2.61911.log.gz 2024-04-24T23:04:42.956 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58871.log 2024-04-24T23:04:42.957 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51674.log 2024-04-24T23:04:42.957 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58871.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58871.log.gz 2024-04-24T23:04:42.962 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51674.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64515.log 2024-04-24T23:04:42.962 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51674.log.gz 2024-04-24T23:04:42.963 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64515.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64515.log.gz 2024-04-24T23:04:42.963 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72354.log 2024-04-24T23:04:42.966 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.72354.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.72354.log.gz 2024-04-24T23:04:42.966 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78711.log 2024-04-24T23:04:42.966 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.78711.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78711.log.gz 2024-04-24T23:04:42.966 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54940.log 2024-04-24T23:04:42.968 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54940.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.54940.log.gz 2024-04-24T23:04:42.968 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64102.log 2024-04-24T23:04:42.969 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.64102.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64102.log.gz 2024-04-24T23:04:42.969 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74083.log 2024-04-24T23:04:42.970 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.74083.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.74083.log.gz 2024-04-24T23:04:42.970 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43459.log 2024-04-24T23:04:42.971 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.43459.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.43459.log.gz 2024-04-24T23:04:42.971 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52821.log 2024-04-24T23:04:42.972 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.52821.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.52821.log.gz 2024-04-24T23:04:42.972 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.48230.log 2024-04-24T23:04:42.978 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.2.48230.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63816.log 2024-04-24T23:04:42.979 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.63816.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.2.48230.log.gz 2024-04-24T23:04:42.979 INFO:teuthology.orchestra.run.smithi134.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63816.log.gz 2024-04-24T23:04:42.979 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58431.log 2024-04-24T23:04:42.979 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57477.log 2024-04-24T23:04:42.990 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58431.log: /var/log/ceph/cluster2-client.mirror.57477.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58431.log.gz 2024-04-24T23:04:42.990 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80691.log 2024-04-24T23:04:42.990 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80691.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80691.log.gz 2024-04-24T23:04:42.990 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56826.log 2024-04-24T23:04:43.000 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56826.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56826.log.gz 2024-04-24T23:04:43.001 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74297.log 2024-04-24T23:04:43.012 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.74297.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74297.log.gz 2024-04-24T23:04:43.012 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64006.log 2024-04-24T23:04:43.012 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.64006.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64006.log.gz 2024-04-24T23:04:43.012 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70311.log 2024-04-24T23:04:43.012 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70311.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70311.log.gz 2024-04-24T23:04:43.012 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72334.log 2024-04-24T23:04:43.012 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72334.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72334.log.gz 2024-04-24T23:04:43.012 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70002.log 2024-04-24T23:04:43.012 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.70002.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70002.log.gz 2024-04-24T23:04:43.012 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50387.log 2024-04-24T23:04:43.012 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.50387.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.50387.log.gz 2024-04-24T23:04:43.013 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66146.log 2024-04-24T23:04:43.013 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66146.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66146.log.gz 2024-04-24T23:04:43.013 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54840.log 2024-04-24T23:04:43.020 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.54840.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.54840.log.gz 2024-04-24T23:04:43.020 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76657.log 2024-04-24T23:04:43.020 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76657.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76657.log.gz 2024-04-24T23:04:43.020 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69305.log 2024-04-24T23:04:43.020 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.69305.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.69305.log.gz 2024-04-24T23:04:43.021 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69264.log 2024-04-24T23:04:43.021 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.69264.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69264.log.gz 2024-04-24T23:04:43.021 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56423.log 2024-04-24T23:04:43.031 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56423.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56423.log.gz 2024-04-24T23:04:43.031 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59135.log 2024-04-24T23:04:43.031 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59135.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59135.log.gz 2024-04-24T23:04:43.031 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52619.log 2024-04-24T23:04:43.031 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.52619.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52619.log.gz 2024-04-24T23:04:43.031 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54055.log 2024-04-24T23:04:43.031 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.54055.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54055.log.gz 2024-04-24T23:04:43.031 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69240.log 2024-04-24T23:04:43.031 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.69240.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69240.log.gz 2024-04-24T23:04:43.032 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52643.log 2024-04-24T23:04:43.032 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.52643.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52643.log.gz 2024-04-24T23:04:43.032 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64404.log 2024-04-24T23:04:43.032 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.64404.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64404.log.gz 2024-04-24T23:04:43.032 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57520.log 2024-04-24T23:04:43.032 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.57520.log: 85.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57520.log.gz 2024-04-24T23:04:43.042 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62943.log 2024-04-24T23:04:43.042 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62943.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62943.log.gz 2024-04-24T23:04:43.042 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71764.log 2024-04-24T23:04:43.042 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.71764.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71764.log.gz 2024-04-24T23:04:43.042 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72217.log 2024-04-24T23:04:43.042 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72217.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72217.log.gz 2024-04-24T23:04:43.042 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77472.log 2024-04-24T23:04:43.042 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77472.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77472.log.gz 2024-04-24T23:04:43.042 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67257.log 2024-04-24T23:04:43.042 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.67257.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.67257.log.gz 2024-04-24T23:04:43.042 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52547.log 2024-04-24T23:04:43.042 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.52547.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52547.log.gz 2024-04-24T23:04:43.042 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52105.log 2024-04-24T23:04:43.053 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.52105.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52105.log.gz 2024-04-24T23:04:43.053 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57274.log 2024-04-24T23:04:43.053 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57274.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57274.log.gz 2024-04-24T23:04:43.053 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70486.log 2024-04-24T23:04:43.053 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.70486.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70486.log.gz 2024-04-24T23:04:43.053 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73600.log 2024-04-24T23:04:43.059 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.73600.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73600.log.gz 2024-04-24T23:04:43.059 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62380.log 2024-04-24T23:04:43.059 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.62380.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.62380.log.gz 2024-04-24T23:04:43.059 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72987.log 2024-04-24T23:04:43.059 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.72987.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.72987.log.gz 2024-04-24T23:04:43.059 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48822.log 2024-04-24T23:04:43.059 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.48822.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.48822.log.gz 2024-04-24T23:04:43.059 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84729.log 2024-04-24T23:04:43.059 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84729.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.84729.log.gz 2024-04-24T23:04:43.060 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82516.log 2024-04-24T23:04:43.060 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.82516.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.82516.log.gz 2024-04-24T23:04:43.070 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69886.log 2024-04-24T23:04:43.070 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69886.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69886.log.gz 2024-04-24T23:04:43.070 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80263.log 2024-04-24T23:04:43.070 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80263.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80263.log.gz 2024-04-24T23:04:43.070 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59677.log 2024-04-24T23:04:43.070 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59677.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59677.log.gz 2024-04-24T23:04:43.070 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70549.log 2024-04-24T23:04:43.070 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70549.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70549.log.gz 2024-04-24T23:04:43.071 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51562.log 2024-04-24T23:04:43.071 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.51562.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51562.log.gz 2024-04-24T23:04:43.071 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61547.log 2024-04-24T23:04:43.075 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61547.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61868.log.gz 2024-04-24T23:04:43.075 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60363.log 2024-04-24T23:04:43.086 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60363.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56443.log 2024-04-24T23:04:43.108 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56443.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56443.log.gz 2024-04-24T23:04:43.108 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.48069.log 2024-04-24T23:04:43.120 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.2.48069.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60450.log 2024-04-24T23:04:43.128 INFO:teuthology.orchestra.run.smithi134.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.2.48069.log.gz 2024-04-24T23:04:43.128 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58007.log.gz 2024-04-24T23:04:43.132 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60450.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52771.log 2024-04-24T23:04:43.152 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52771.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.81922.log.gz 2024-04-24T23:04:43.152 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79813.log 2024-04-24T23:04:43.163 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79813.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.79813.log.gz 2024-04-24T23:04:43.163 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59924.log 2024-04-24T23:04:43.188 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59924.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53983.log 2024-04-24T23:04:43.198 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.53983.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54007.log 2024-04-24T23:04:43.199 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53983.log.gz 2024-04-24T23:04:43.210 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.54007.log: gzip -5 88.8%--verbose -- -- replaced with /var/log/ceph/cluster1-client.mirror.54007.log.gz/var/log/ceph/cluster1-client.mirror.68829.log 2024-04-24T23:04:43.210 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T23:04:43.221 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.68829.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49214.log 2024-04-24T23:04:43.222 INFO:teuthology.orchestra.run.smithi134.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68829.log.gz 2024-04-24T23:04:43.234 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49214.log: 54.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49214.log.gz 2024-04-24T23:04:43.245 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53063.log 2024-04-24T23:04:43.256 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53063.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57054.log 2024-04-24T23:04:43.256 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53063.log.gz 2024-04-24T23:04:43.267 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57054.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57455.log 2024-04-24T23:04:43.270 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57477.log.gz 2024-04-24T23:04:43.271 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57455.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57455.log.gz 2024-04-24T23:04:43.271 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58623.log 2024-04-24T23:04:43.273 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58623.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58623.log.gz 2024-04-24T23:04:43.273 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82580.log 2024-04-24T23:04:43.274 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82580.log: 86.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82580.log.gz 2024-04-24T23:04:43.274 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59215.log 2024-04-24T23:04:43.275 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59215.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59215.log.gz 2024-04-24T23:04:43.275 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70608.log 2024-04-24T23:04:43.277 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70608.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70608.log.gz 2024-04-24T23:04:43.277 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54379.log 2024-04-24T23:04:43.279 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54379.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.3.48090.log.gz 2024-04-24T23:04:43.279 INFO:teuthology.orchestra.run.smithi134.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54379.log.gz 2024-04-24T23:04:43.279 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52902.log 2024-04-24T23:04:43.280 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52902.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.52902.log.gz 2024-04-24T23:04:43.280 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54446.log 2024-04-24T23:04:43.281 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54446.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.54446.log.gz 2024-04-24T23:04:43.282 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72943.log 2024-04-24T23:04:43.283 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72943.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72943.log.gz 2024-04-24T23:04:43.283 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76008.log 2024-04-24T23:04:43.284 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76008.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76008.log.gz 2024-04-24T23:04:43.284 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71637.log 2024-04-24T23:04:43.296 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71637.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71637.log.gz 2024-04-24T23:04:43.296 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51071.log 2024-04-24T23:04:43.297 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61547.log.gz 2024-04-24T23:04:43.313 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.51071.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51071.log.gz 2024-04-24T23:04:43.313 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60428.log 2024-04-24T23:04:43.316 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59719.log.gz 2024-04-24T23:04:43.317 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79948.log 2024-04-24T23:04:43.317 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60428.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60428.log.gz 2024-04-24T23:04:43.318 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79948.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.79948.log.gz 2024-04-24T23:04:43.318 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59882.log 2024-04-24T23:04:43.318 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74655.log 2024-04-24T23:04:43.319 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59882.log: /var/log/ceph/cluster1-client.mirror.74655.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59882.log.gz 2024-04-24T23:04:43.319 INFO:teuthology.orchestra.run.smithi134.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74655.log.gz 2024-04-24T23:04:43.319 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82166.log 2024-04-24T23:04:43.320 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67279.log 2024-04-24T23:04:43.321 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82166.log: /var/log/ceph/cluster2-client.mirror.67279.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67279.log.gz 2024-04-24T23:04:43.321 INFO:teuthology.orchestra.run.smithi134.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82166.log.gz 2024-04-24T23:04:43.321 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63094.log 2024-04-24T23:04:43.322 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.63094.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79206.log 2024-04-24T23:04:43.322 INFO:teuthology.orchestra.run.smithi134.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63094.log.gz 2024-04-24T23:04:43.323 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79206.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.79206.log.gz 2024-04-24T23:04:43.323 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74893.log 2024-04-24T23:04:43.324 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.74893.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74893.log.gz 2024-04-24T23:04:43.324 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57316.log 2024-04-24T23:04:43.328 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57316.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83926.log 2024-04-24T23:04:43.331 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83926.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.83926.log.gz 2024-04-24T23:04:43.331 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73862.log 2024-04-24T23:04:43.334 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.73862.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.73862.log.gz 2024-04-24T23:04:43.334 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83583.log 2024-04-24T23:04:43.344 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83583.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54494.log 2024-04-24T23:04:43.349 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60450.log.gz 2024-04-24T23:04:43.351 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.54494.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54494.log.gz 2024-04-24T23:04:43.351 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60573.log 2024-04-24T23:04:43.352 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.60573.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60573.log.gz 2024-04-24T23:04:43.352 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76637.log 2024-04-24T23:04:43.353 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76637.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76637.log.gz 2024-04-24T23:04:43.353 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53704.log 2024-04-24T23:04:43.359 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.53704.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53704.log.gz 2024-04-24T23:04:43.359 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66880.log 2024-04-24T23:04:43.360 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66880.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66880.log.gz 2024-04-24T23:04:43.361 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62808.log 2024-04-24T23:04:43.362 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.62808.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62808.log.gz 2024-04-24T23:04:43.362 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69148.log 2024-04-24T23:04:43.364 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69148.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69148.log.gz 2024-04-24T23:04:43.364 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72681.log 2024-04-24T23:04:43.365 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72681.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72681.log.gz 2024-04-24T23:04:43.365 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63724.log 2024-04-24T23:04:43.367 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63724.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63724.log.gz 2024-04-24T23:04:43.367 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62642.log 2024-04-24T23:04:43.368 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.62642.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62642.log.gz 2024-04-24T23:04:43.368 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.48116.log 2024-04-24T23:04:43.379 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.0.48116.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64832.log 2024-04-24T23:04:43.379 INFO:teuthology.orchestra.run.smithi134.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.48116.log.gz 2024-04-24T23:04:43.381 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64832.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64832.log.gz 2024-04-24T23:04:43.381 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74679.log 2024-04-24T23:04:43.382 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.74679.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.74679.log.gz 2024-04-24T23:04:43.382 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76802.log 2024-04-24T23:04:43.384 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76802.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76802.log.gz 2024-04-24T23:04:43.384 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78801.log 2024-04-24T23:04:43.385 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.78801.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78801.log.gz 2024-04-24T23:04:43.385 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71812.log 2024-04-24T23:04:43.386 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.71812.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71812.log.gz 2024-04-24T23:04:43.386 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68478.log 2024-04-24T23:04:43.387 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.68478.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.68478.log.gz 2024-04-24T23:04:43.387 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62206.log 2024-04-24T23:04:43.388 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.62206.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62206.log.gz 2024-04-24T23:04:43.388 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57626.log 2024-04-24T23:04:43.390 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57626.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57626.log.gz 2024-04-24T23:04:43.390 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69692.log 2024-04-24T23:04:43.391 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.69692.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69692.log.gz 2024-04-24T23:04:43.391 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81757.log 2024-04-24T23:04:43.392 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.81757.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81757.log.gz 2024-04-24T23:04:43.392 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84277.log 2024-04-24T23:04:43.394 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.84277.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.84277.log.gz 2024-04-24T23:04:43.394 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53747.log 2024-04-24T23:04:43.400 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.53747.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53747.log.gz 2024-04-24T23:04:43.400 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82301.log 2024-04-24T23:04:43.401 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82301.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.82301.log.gz 2024-04-24T23:04:43.401 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80083.log 2024-04-24T23:04:43.402 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80083.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.80083.log.gz 2024-04-24T23:04:43.402 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67495.log 2024-04-24T23:04:43.405 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.67495.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.67495.log.gz 2024-04-24T23:04:43.405 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73342.log 2024-04-24T23:04:43.407 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73342.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73342.log.gz 2024-04-24T23:04:43.407 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61067.log 2024-04-24T23:04:43.408 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61067.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61067.log.gz 2024-04-24T23:04:43.408 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65638.log 2024-04-24T23:04:43.410 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65638.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65638.log.gz 2024-04-24T23:04:43.410 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67869.log 2024-04-24T23:04:43.412 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.67869.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52149.log.gz 2024-04-24T23:04:43.412 INFO:teuthology.orchestra.run.smithi134.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.67869.log.gz 2024-04-24T23:04:43.413 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53935.log 2024-04-24T23:04:43.413 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53935.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53935.log.gz 2024-04-24T23:04:43.413 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78378.log 2024-04-24T23:04:43.416 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78378.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.78378.log.gz 2024-04-24T23:04:43.416 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58414.log 2024-04-24T23:04:43.417 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58414.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58414.log.gz 2024-04-24T23:04:43.417 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51716.log 2024-04-24T23:04:43.418 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51716.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51716.log.gz 2024-04-24T23:04:43.419 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77431.log 2024-04-24T23:04:43.420 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77431.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77431.log.gz 2024-04-24T23:04:43.420 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54426.log 2024-04-24T23:04:43.422 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54426.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.54426.log.gz 2024-04-24T23:04:43.422 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69478.log 2024-04-24T23:04:43.422 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.69478.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69478.log.gz 2024-04-24T23:04:43.423 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47563.log 2024-04-24T23:04:43.423 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.47563.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.47563.log.gz 2024-04-24T23:04:43.423 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79836.log 2024-04-24T23:04:43.424 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79836.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79836.log.gz 2024-04-24T23:04:43.424 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57294.log 2024-04-24T23:04:43.425 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57294.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57294.log.gz 2024-04-24T23:04:43.425 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51881.log 2024-04-24T23:04:43.427 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51881.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51881.log.gz 2024-04-24T23:04:43.427 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82537.log 2024-04-24T23:04:43.429 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82537.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.82537.log.gz 2024-04-24T23:04:43.429 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82793.log 2024-04-24T23:04:43.430 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82793.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.82793.log.gz 2024-04-24T23:04:43.430 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.64315.log 2024-04-24T23:04:43.435 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.0.64315.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47612.log 2024-04-24T23:04:43.435 INFO:teuthology.orchestra.run.smithi080.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-04-24T23:04:43.437 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.47612.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.47612.log.gz 2024-04-24T23:04:43.445 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84957.log 2024-04-24T23:04:43.447 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.84957.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84957.log.gz 2024-04-24T23:04:43.447 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70700.log 2024-04-24T23:04:43.448 INFO:teuthology.orchestra.run.smithi134.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.0.64315.log.gz 2024-04-24T23:04:43.449 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73059.log 2024-04-24T23:04:43.449 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.70700.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70700.log.gz 2024-04-24T23:04:43.450 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81616.log 2024-04-24T23:04:43.450 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.73059.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73059.log.gz 2024-04-24T23:04:43.452 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81616.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.81616.log.gz 2024-04-24T23:04:43.452 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74917.log 2024-04-24T23:04:43.453 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.74917.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74917.log.gz 2024-04-24T23:04:43.453 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53403.log 2024-04-24T23:04:43.455 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71265.log 2024-04-24T23:04:43.455 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53403.log: /var/log/ceph/cluster1-client.mirror.71265.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53403.log.gz 2024-04-24T23:04:43.456 INFO:teuthology.orchestra.run.smithi134.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.71265.log.gz 2024-04-24T23:04:43.456 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49129.log 2024-04-24T23:04:43.456 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49129.log: 53.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49129.log.gz 2024-04-24T23:04:43.457 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53423.log 2024-04-24T23:04:43.457 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53423.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53423.log.gz 2024-04-24T23:04:43.457 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55429.log 2024-04-24T23:04:43.463 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.55429.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56062.log 2024-04-24T23:04:43.468 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56062.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55429.log.gz 2024-04-24T23:04:43.468 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75297.log 2024-04-24T23:04:43.468 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56062.log.gz 2024-04-24T23:04:43.471 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72197.log 2024-04-24T23:04:43.471 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75297.log: /var/log/ceph/cluster2-client.mirror.72197.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72197.log.gz 2024-04-24T23:04:43.472 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66999.log 2024-04-24T23:04:43.472 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66999.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75297.log.gz 2024-04-24T23:04:43.473 INFO:teuthology.orchestra.run.smithi134.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66999.log.gz 2024-04-24T23:04:43.473 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85057.log 2024-04-24T23:04:43.474 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.85057.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.85057.log.gz 2024-04-24T23:04:43.474 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82856.log 2024-04-24T23:04:43.475 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.82856.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.82856.log.gz 2024-04-24T23:04:43.475 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78068.log 2024-04-24T23:04:43.478 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78068.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.78068.log.gz 2024-04-24T23:04:43.478 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50847.log 2024-04-24T23:04:43.479 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58112.log 2024-04-24T23:04:43.479 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50847.log: /var/log/ceph/cluster2-client.mirror.58112.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58112.log.gz 2024-04-24T23:04:43.479 INFO:teuthology.orchestra.run.smithi134.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50847.log.gz 2024-04-24T23:04:43.479 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85995.log 2024-04-24T23:04:43.480 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.85995.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85995.log.gz 2024-04-24T23:04:43.480 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68930.log 2024-04-24T23:04:43.481 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78756.log 2024-04-24T23:04:43.481 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68930.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68930.log.gz 2024-04-24T23:04:43.482 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.78756.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78756.log.gz 2024-04-24T23:04:43.482 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82145.log 2024-04-24T23:04:43.483 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46779.log 2024-04-24T23:04:43.483 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.82145.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.82145.log.gz 2024-04-24T23:04:43.483 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.46779.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.46779.log.gz 2024-04-24T23:04:43.483 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70724.log 2024-04-24T23:04:43.484 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77596.log 2024-04-24T23:04:43.487 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.70724.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.70724.log.gz 2024-04-24T23:04:43.487 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77596.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77596.log.gz 2024-04-24T23:04:43.487 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66190.log 2024-04-24T23:04:43.488 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.66190.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.66190.log.gz 2024-04-24T23:04:43.488 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85556.log 2024-04-24T23:04:43.489 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.85556.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85556.log.gz 2024-04-24T23:04:43.489 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73580.log 2024-04-24T23:04:43.491 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73580.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73580.log.gz 2024-04-24T23:04:43.491 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70043.log 2024-04-24T23:04:43.493 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.70043.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.70043.log.gz 2024-04-24T23:04:43.493 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73987.log 2024-04-24T23:04:43.494 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.73987.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73987.log.gz 2024-04-24T23:04:43.494 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54867.log 2024-04-24T23:04:43.495 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.54867.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.54867.log.gz 2024-04-24T23:04:43.495 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51631.log 2024-04-24T23:04:43.499 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51631.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47367.log 2024-04-24T23:04:43.501 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51631.log.gz 2024-04-24T23:04:43.502 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.47367.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.47367.log.gz 2024-04-24T23:04:43.502 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70023.log 2024-04-24T23:04:43.504 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70023.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70023.log.gz 2024-04-24T23:04:43.504 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56543.log 2024-04-24T23:04:43.505 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56543.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56543.log.gz 2024-04-24T23:04:43.505 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53106.log 2024-04-24T23:04:43.512 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53106.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53106.log.gz 2024-04-24T23:04:43.512 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62308.log 2024-04-24T23:04:43.512 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62308.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62308.log.gz 2024-04-24T23:04:43.513 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63792.log 2024-04-24T23:04:43.513 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.63792.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63792.log.gz 2024-04-24T23:04:43.513 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56403.log 2024-04-24T23:04:43.515 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56403.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56403.log.gz 2024-04-24T23:04:43.515 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55409.log 2024-04-24T23:04:43.520 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55409.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55409.log.gz 2024-04-24T23:04:43.521 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46483.log 2024-04-24T23:04:43.521 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.46483.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73487.log 2024-04-24T23:04:43.521 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.46483.log.gz 2024-04-24T23:04:43.522 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.73487.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73487.log.gz 2024-04-24T23:04:43.522 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60391.log 2024-04-24T23:04:43.523 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60391.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60391.log.gz 2024-04-24T23:04:43.523 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72265.log 2024-04-24T23:04:43.524 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.72265.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.72265.log.gz 2024-04-24T23:04:43.524 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55558.log 2024-04-24T23:04:43.530 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.55558.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59198.log 2024-04-24T23:04:43.530 INFO:teuthology.orchestra.run.smithi134.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55558.log.gz 2024-04-24T23:04:43.531 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59198.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59198.log.gz 2024-04-24T23:04:43.531 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57418.log 2024-04-24T23:04:43.531 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57418.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57418.log.gz 2024-04-24T23:04:43.531 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65819.log 2024-04-24T23:04:43.534 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.65819.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65819.log.gz 2024-04-24T23:04:43.534 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75917.log 2024-04-24T23:04:43.536 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75917.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75917.log.gz 2024-04-24T23:04:43.537 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54749.log 2024-04-24T23:04:43.537 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.54749.log: 81.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54749.log.gz 2024-04-24T23:04:43.537 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79093.log 2024-04-24T23:04:43.538 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79093.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79093.log.gz 2024-04-24T23:04:43.538 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70914.log 2024-04-24T23:04:43.539 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.70914.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70914.log.gz 2024-04-24T23:04:43.539 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55933.log 2024-04-24T23:04:43.545 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55933.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54096.log 2024-04-24T23:04:43.545 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.54096.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60363.log.gz 2024-04-24T23:04:43.545 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55933.log.gz 2024-04-24T23:04:43.549 INFO:teuthology.orchestra.run.smithi134.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54096.log.gz 2024-04-24T23:04:43.549 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71574.log 2024-04-24T23:04:43.549 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79453.log 2024-04-24T23:04:43.550 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.71574.log: /var/log/ceph/cluster1-client.mirror.79453.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71574.log.gz 2024-04-24T23:04:43.550 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.79453.log.gz 2024-04-24T23:04:43.550 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50671.log 2024-04-24T23:04:43.551 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77071.log 2024-04-24T23:04:43.551 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50671.log: /var/log/ceph/cluster2-client.mirror.77071.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77071.log.gz 2024-04-24T23:04:43.551 INFO:teuthology.orchestra.run.smithi134.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50671.log.gz 2024-04-24T23:04:43.551 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76905.log 2024-04-24T23:04:43.552 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85082.log 2024-04-24T23:04:43.552 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76905.log: /var/log/ceph/cluster2-client.mirror.85082.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.85082.log.gz 2024-04-24T23:04:43.552 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61747.log 2024-04-24T23:04:43.555 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72821.log 2024-04-24T23:04:43.555 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61747.log: /var/log/ceph/cluster1-client.mirror.72821.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.72821.log.gz 2024-04-24T23:04:43.555 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71526.log 2024-04-24T23:04:43.555 INFO:teuthology.orchestra.run.smithi134.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61747.log.gz 2024-04-24T23:04:43.555 INFO:teuthology.orchestra.run.smithi134.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76905.log.gz 2024-04-24T23:04:43.555 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.71526.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71526.log.gz 2024-04-24T23:04:43.555 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54353.log 2024-04-24T23:04:43.556 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.48004.log 2024-04-24T23:04:43.556 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.54353.log: /var/log/ceph/cluster1-client.mirror.0.48004.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.54353.log.gz 2024-04-24T23:04:43.556 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47710.log 2024-04-24T23:04:43.556 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.47710.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.47710.log.gz 2024-04-24T23:04:43.556 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76153.log 2024-04-24T23:04:43.558 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60979.log 2024-04-24T23:04:43.558 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76153.log: /var/log/ceph/cluster2-client.mirror.60979.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76153.log.gz 2024-04-24T23:04:43.558 INFO:teuthology.orchestra.run.smithi134.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60979.log.gz 2024-04-24T23:04:43.558 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71390.log 2024-04-24T23:04:43.559 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72289.log 2024-04-24T23:04:43.559 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.71390.log: /var/log/ceph/cluster1-client.mirror.72289.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71390.log.gz 2024-04-24T23:04:43.559 INFO:teuthology.orchestra.run.smithi134.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.72289.log.gz 2024-04-24T23:04:43.559 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72797.log 2024-04-24T23:04:43.560 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61016.log 2024-04-24T23:04:43.560 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.72797.log: /var/log/ceph/cluster2-client.mirror.61016.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.72797.log.gz 2024-04-24T23:04:43.560 INFO:teuthology.orchestra.run.smithi134.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61016.log.gz 2024-04-24T23:04:43.561 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76988.log 2024-04-24T23:04:43.562 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.48004.log.gz 2024-04-24T23:04:43.562 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67041.log 2024-04-24T23:04:43.562 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76988.log: /var/log/ceph/cluster2-client.mirror.67041.log: gzip -5 89.3%--verbose -- -- replaced with /var/log/ceph/cluster2-client.mirror.76988.log.gz/var/log/ceph/cluster2-client.mirror.58287.log 2024-04-24T23:04:43.562 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T23:04:43.562 INFO:teuthology.orchestra.run.smithi134.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67041.log.gz 2024-04-24T23:04:43.563 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58287.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68333.log 2024-04-24T23:04:43.565 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68333.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68333.log.gz 2024-04-24T23:04:43.565 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81863.log 2024-04-24T23:04:43.566 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81863.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.81863.log.gz 2024-04-24T23:04:43.566 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55824.log 2024-04-24T23:04:43.572 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64856.log 2024-04-24T23:04:43.572 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.55824.log: /var/log/ceph/cluster1-client.mirror.64856.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64856.log.gz 2024-04-24T23:04:43.572 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74962.log 2024-04-24T23:04:43.572 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74962.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74962.log.gz 2024-04-24T23:04:43.572 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84863.log 2024-04-24T23:04:43.572 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84863.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.84863.log.gz 2024-04-24T23:04:43.573 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50800.log 2024-04-24T23:04:43.573 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50800.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55824.log.gz 2024-04-24T23:04:43.573 INFO:teuthology.orchestra.run.smithi134.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50800.log.gz 2024-04-24T23:04:43.573 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82218.log 2024-04-24T23:04:43.573 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.82218.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.82218.log.gz 2024-04-24T23:04:43.573 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58749.log 2024-04-24T23:04:43.574 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58749.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58749.log.gz 2024-04-24T23:04:43.574 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83503.log 2024-04-24T23:04:43.577 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83503.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.83503.log.gz 2024-04-24T23:04:43.577 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59314.log 2024-04-24T23:04:43.578 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59314.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59314.log.gz 2024-04-24T23:04:43.578 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76111.log 2024-04-24T23:04:43.580 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76111.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84190.log 2024-04-24T23:04:43.581 INFO:teuthology.orchestra.run.smithi134.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76111.log.gz 2024-04-24T23:04:43.582 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84190.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.84190.log.gz 2024-04-24T23:04:43.582 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66860.log 2024-04-24T23:04:43.584 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66860.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66860.log.gz 2024-04-24T23:04:43.584 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74059.log 2024-04-24T23:04:43.585 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.74059.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74059.log.gz 2024-04-24T23:04:43.585 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54183.log 2024-04-24T23:04:43.590 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54183.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56881.log 2024-04-24T23:04:43.591 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56881.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56881.log.gz 2024-04-24T23:04:43.592 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85850.log 2024-04-24T23:04:43.592 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.85850.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85850.log.gz 2024-04-24T23:04:43.593 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66384.log 2024-04-24T23:04:43.594 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66384.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66384.log.gz 2024-04-24T23:04:43.594 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81164.log 2024-04-24T23:04:43.595 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81164.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.81164.log.gz 2024-04-24T23:04:43.595 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.52387.log 2024-04-24T23:04:43.606 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.3.52387.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52946.log 2024-04-24T23:04:43.607 INFO:teuthology.orchestra.run.smithi134.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.3.52387.log.gz 2024-04-24T23:04:43.608 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.52946.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52946.log.gz 2024-04-24T23:04:43.608 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61488.log 2024-04-24T23:04:43.609 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61488.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61488.log.gz 2024-04-24T23:04:43.609 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59336.log 2024-04-24T23:04:43.620 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59336.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58771.log 2024-04-24T23:04:43.646 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58771.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82260.log 2024-04-24T23:04:43.657 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52771.log.gz 2024-04-24T23:04:43.657 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82260.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66262.log 2024-04-24T23:04:43.657 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82260.log.gz 2024-04-24T23:04:43.677 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.66262.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66262.log.gz 2024-04-24T23:04:43.677 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78234.log 2024-04-24T23:04:43.700 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78234.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69954.log 2024-04-24T23:04:43.711 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78234.log.gz 2024-04-24T23:04:43.722 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.69954.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69954.log.gz 2024-04-24T23:04:43.722 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50608.log 2024-04-24T23:04:43.733 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.50608.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50608.log.gz 2024-04-24T23:04:43.733 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59924.log.gz 2024-04-24T23:04:43.733 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79296.log 2024-04-24T23:04:43.744 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79296.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.79296.log.gz 2024-04-24T23:04:43.744 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47075.log 2024-04-24T23:04:43.744 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.47075.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.47075.log.gz 2024-04-24T23:04:43.744 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56343.log 2024-04-24T23:04:43.744 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56343.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56343.log.gz 2024-04-24T23:04:43.744 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77196.log 2024-04-24T23:04:43.744 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77196.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77196.log.gz 2024-04-24T23:04:43.744 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78733.log 2024-04-24T23:04:43.744 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.78733.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78733.log.gz 2024-04-24T23:04:43.744 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61789.log 2024-04-24T23:04:43.755 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61789.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49180.log 2024-04-24T23:04:43.764 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49180.log: 57.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49180.log.gz 2024-04-24T23:04:43.764 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.83583.log.gz 2024-04-24T23:04:43.765 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49991.log 2024-04-24T23:04:43.776 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49991.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68550.log 2024-04-24T23:04:43.800 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.68550.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64143.log 2024-04-24T23:04:43.800 INFO:teuthology.orchestra.run.smithi134.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.68550.log.gz 2024-04-24T23:04:43.825 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.64143.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64143.log.gz 2024-04-24T23:04:43.825 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50316.log 2024-04-24T23:04:43.836 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.50316.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.50316.log.gz 2024-04-24T23:04:43.836 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63260.log 2024-04-24T23:04:43.855 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63260.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49290.log 2024-04-24T23:04:43.855 INFO:teuthology.orchestra.run.smithi134.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63260.log.gz 2024-04-24T23:04:43.866 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58287.log.gz 2024-04-24T23:04:43.866 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49290.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49290.log.gz 2024-04-24T23:04:43.866 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71696.log 2024-04-24T23:04:43.866 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71696.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71696.log.gz 2024-04-24T23:04:43.866 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66904.log 2024-04-24T23:04:43.866 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.66904.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.66904.log.gz 2024-04-24T23:04:43.866 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80151.log 2024-04-24T23:04:43.866 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80151.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.80151.log.gz 2024-04-24T23:04:43.866 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55652.log 2024-04-24T23:04:43.877 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.55652.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55652.log.gz 2024-04-24T23:04:43.877 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78399.log 2024-04-24T23:04:43.877 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78399.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.78399.log.gz 2024-04-24T23:04:43.877 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78958.log 2024-04-24T23:04:43.877 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.78958.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78958.log.gz 2024-04-24T23:04:43.877 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72051.log 2024-04-24T23:04:43.877 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.72051.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.72051.log.gz 2024-04-24T23:04:43.877 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74273.log 2024-04-24T23:04:43.877 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.74273.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74273.log.gz 2024-04-24T23:04:43.877 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82900.log 2024-04-24T23:04:43.888 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82900.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.82900.log.gz 2024-04-24T23:04:43.888 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75814.log 2024-04-24T23:04:43.888 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75814.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75814.log.gz 2024-04-24T23:04:43.888 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70765.log 2024-04-24T23:04:43.888 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.70765.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70765.log.gz 2024-04-24T23:04:43.888 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82952.log 2024-04-24T23:04:43.888 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.82952.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.82952.log.gz 2024-04-24T23:04:43.888 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76350.log 2024-04-24T23:04:43.888 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76350.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76350.log.gz 2024-04-24T23:04:43.888 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59257.log 2024-04-24T23:04:43.888 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59257.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68312.log 2024-04-24T23:04:43.899 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57054.log.gz 2024-04-24T23:04:43.899 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.68312.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68312.log.gz 2024-04-24T23:04:43.899 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84450.log 2024-04-24T23:04:43.904 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84450.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.84450.log.gz 2024-04-24T23:04:43.904 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57948.log 2024-04-24T23:04:43.904 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57948.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57948.log.gz 2024-04-24T23:04:43.904 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65278.log 2024-04-24T23:04:43.904 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.65278.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.65278.log.gz 2024-04-24T23:04:43.904 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61525.log 2024-04-24T23:04:43.904 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61525.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61525.log.gz 2024-04-24T23:04:43.904 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58828.log 2024-04-24T23:04:43.904 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58828.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58828.log.gz 2024-04-24T23:04:43.904 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63861.log 2024-04-24T23:04:43.915 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63861.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63861.log.gz 2024-04-24T23:04:43.915 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75380.log 2024-04-24T23:04:43.915 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75380.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75380.log.gz 2024-04-24T23:04:43.915 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75318.log 2024-04-24T23:04:43.915 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75318.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75318.log.gz 2024-04-24T23:04:43.915 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82755.log 2024-04-24T23:04:43.915 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82755.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82755.log.gz 2024-04-24T23:04:43.915 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61730.log 2024-04-24T23:04:43.915 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61730.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61730.log.gz 2024-04-24T23:04:43.915 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57215.log 2024-04-24T23:04:43.915 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57215.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57215.log.gz 2024-04-24T23:04:43.915 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56039.log 2024-04-24T23:04:43.918 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.56039.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58771.log.gz 2024-04-24T23:04:43.918 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56039.log.gz 2024-04-24T23:04:43.918 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79521.log 2024-04-24T23:04:43.919 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79521.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.79521.log.gz 2024-04-24T23:04:43.919 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80353.log 2024-04-24T23:04:43.920 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80353.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80353.log.gz 2024-04-24T23:04:43.920 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79228.log 2024-04-24T23:04:43.921 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79228.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79228.log.gz 2024-04-24T23:04:43.921 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80038.log 2024-04-24T23:04:43.922 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58376.log 2024-04-24T23:04:43.922 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80038.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.80038.log.gz 2024-04-24T23:04:43.925 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58376.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57316.log.gz 2024-04-24T23:04:43.927 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49231.log 2024-04-24T23:04:43.927 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49231.log: 54.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49231.log.gz 2024-04-24T23:04:43.928 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57398.log 2024-04-24T23:04:43.939 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57398.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57965.log 2024-04-24T23:04:43.940 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57965.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57965.log.gz 2024-04-24T23:04:43.942 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61789.log.gz 2024-04-24T23:04:43.950 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79071.log 2024-04-24T23:04:43.950 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71224.log 2024-04-24T23:04:43.950 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79071.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79071.log.gz 2024-04-24T23:04:43.962 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61709.log 2024-04-24T23:04:43.962 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.71224.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.71224.log.gz 2024-04-24T23:04:43.962 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61709.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63478.log 2024-04-24T23:04:43.964 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63478.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63478.log.gz 2024-04-24T23:04:43.964 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58850.log 2024-04-24T23:04:43.975 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58850.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67168.log 2024-04-24T23:04:43.990 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.67168.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.67168.log.gz 2024-04-24T23:04:43.990 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58184.log 2024-04-24T23:04:44.010 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75690.log 2024-04-24T23:04:44.020 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58184.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58184.log.gz 2024-04-24T23:04:44.034 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75690.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83117.log 2024-04-24T23:04:44.035 INFO:teuthology.orchestra.run.smithi134.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75690.log.gz 2024-04-24T23:04:44.046 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.83117.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.83117.log.gz 2024-04-24T23:04:44.046 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78576.log 2024-04-24T23:04:44.054 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.78576.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78576.log.gz 2024-04-24T23:04:44.054 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69716.log 2024-04-24T23:04:44.065 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.69716.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69716.log.gz 2024-04-24T23:04:44.065 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70269.log 2024-04-24T23:04:44.089 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70269.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68613.log 2024-04-24T23:04:44.096 INFO:teuthology.orchestra.run.smithi134.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70269.log.gz 2024-04-24T23:04:44.108 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68613.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75876.log 2024-04-24T23:04:44.108 INFO:teuthology.orchestra.run.smithi134.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68613.log.gz 2024-04-24T23:04:44.118 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54183.log.gz 2024-04-24T23:04:44.119 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75876.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75876.log.gz 2024-04-24T23:04:44.119 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61038.log 2024-04-24T23:04:44.130 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61038.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68571.log 2024-04-24T23:04:44.148 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68571.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68571.log.gz 2024-04-24T23:04:44.148 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83541.log 2024-04-24T23:04:44.159 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83541.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58451.log 2024-04-24T23:04:44.159 INFO:teuthology.orchestra.run.smithi134.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.83541.log.gz 2024-04-24T23:04:44.170 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58451.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58451.log.gz 2024-04-24T23:04:44.170 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84111.log 2024-04-24T23:04:44.181 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84111.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78192.log 2024-04-24T23:04:44.181 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84111.log.gz 2024-04-24T23:04:44.193 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78192.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81123.log 2024-04-24T23:04:44.200 INFO:teuthology.orchestra.run.smithi134.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.78192.log.gz 2024-04-24T23:04:44.211 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81123.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81123.log.gz 2024-04-24T23:04:44.211 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66928.log 2024-04-24T23:04:44.222 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.66928.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66928.log.gz 2024-04-24T23:04:44.222 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63070.log 2024-04-24T23:04:44.223 INFO:teuthology.orchestra.run.smithi134.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49991.log.gz 2024-04-24T23:04:44.231 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59336.log.gz 2024-04-24T23:04:44.231 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79633.log 2024-04-24T23:04:44.232 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.63070.log: /var/log/ceph/cluster1-client.mirror.79633.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63070.log.gz 2024-04-24T23:04:44.232 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75132.log 2024-04-24T23:04:44.232 INFO:teuthology.orchestra.run.smithi134.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.79633.log.gz 2024-04-24T23:04:44.234 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73884.log 2024-04-24T23:04:44.234 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75132.log: /var/log/ceph/cluster2-client.mirror.73884.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75132.log.gz 2024-04-24T23:04:44.234 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56363.log 2024-04-24T23:04:44.235 INFO:teuthology.orchestra.run.smithi134.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73884.log.gz 2024-04-24T23:04:44.235 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72003.log 2024-04-24T23:04:44.235 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56363.log: /var/log/ceph/cluster1-client.mirror.72003.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56363.log.gz 2024-04-24T23:04:44.235 INFO:teuthology.orchestra.run.smithi134.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.72003.log.gz 2024-04-24T23:04:44.235 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53042.log 2024-04-24T23:04:44.236 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57826.log 2024-04-24T23:04:44.237 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.53042.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53042.log.gz 2024-04-24T23:04:44.237 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57826.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61588.log 2024-04-24T23:04:44.237 INFO:teuthology.orchestra.run.smithi134.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57826.log.gz 2024-04-24T23:04:44.237 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61588.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61588.log.gz 2024-04-24T23:04:44.238 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77029.log 2024-04-24T23:04:44.239 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72559.log 2024-04-24T23:04:44.240 INFO:teuthology.orchestra.run.smithi134.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77029.log.gz 2024-04-24T23:04:44.240 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.72559.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.72559.log.gz 2024-04-24T23:04:44.240 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73011.log 2024-04-24T23:04:44.241 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.73011.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73011.log.gz 2024-04-24T23:04:44.241 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56019.log 2024-04-24T23:04:44.245 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56019.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56019.log.gz 2024-04-24T23:04:44.245 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56503.log 2024-04-24T23:04:44.246 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56503.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56503.log.gz 2024-04-24T23:04:44.247 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53769.log 2024-04-24T23:04:44.248 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53769.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53769.log.gz 2024-04-24T23:04:44.248 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48795.log 2024-04-24T23:04:44.248 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.48795.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.48795.log.gz 2024-04-24T23:04:44.248 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.48253.log 2024-04-24T23:04:44.250 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.3.48253.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57985.log 2024-04-24T23:04:44.251 INFO:teuthology.orchestra.run.smithi134.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.3.48253.log.gz 2024-04-24T23:04:44.252 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57985.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57985.log.gz 2024-04-24T23:04:44.252 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65492.log 2024-04-24T23:04:44.253 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.65492.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65492.log.gz 2024-04-24T23:04:44.253 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85654.log 2024-04-24T23:04:44.253 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.85654.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85654.log.gz 2024-04-24T23:04:44.253 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64928.log 2024-04-24T23:04:44.254 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.64928.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64928.log.gz 2024-04-24T23:04:44.254 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82280.log 2024-04-24T23:04:44.256 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82280.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82280.log.gz 2024-04-24T23:04:44.256 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60488.log 2024-04-24T23:04:44.257 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60488.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60488.log.gz 2024-04-24T23:04:44.258 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47906.log 2024-04-24T23:04:44.258 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.47906.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.47906.log.gz 2024-04-24T23:04:44.258 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.37981.log 2024-04-24T23:04:44.259 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph.tmp-client.admin.37981.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.37981.log.gz 2024-04-24T23:04:44.259 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46828.log 2024-04-24T23:04:44.259 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.46828.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.46828.log.gz 2024-04-24T23:04:44.260 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60917.log 2024-04-24T23:04:44.261 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60917.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60917.log.gz 2024-04-24T23:04:44.261 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75421.log 2024-04-24T23:04:44.262 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75421.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64904.log 2024-04-24T23:04:44.263 INFO:teuthology.orchestra.run.smithi134.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75421.log.gz 2024-04-24T23:04:44.263 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.64904.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.64904.log.gz 2024-04-24T23:04:44.264 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64666.log 2024-04-24T23:04:44.264 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77986.log 2024-04-24T23:04:44.265 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.64666.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64666.log.gz 2024-04-24T23:04:44.265 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77986.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.77986.log.gz 2024-04-24T23:04:44.265 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79858.log 2024-04-24T23:04:44.266 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73205.log 2024-04-24T23:04:44.267 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79858.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.79858.log.gz 2024-04-24T23:04:44.267 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73205.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73205.log.gz 2024-04-24T23:04:44.267 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52595.log 2024-04-24T23:04:44.268 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.52595.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.52595.log.gz 2024-04-24T23:04:44.268 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54470.log 2024-04-24T23:04:44.268 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.54470.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54470.log.gz 2024-04-24T23:04:44.269 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81900.log 2024-04-24T23:04:44.269 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81900.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.81900.log.gz 2024-04-24T23:04:44.269 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48229.log 2024-04-24T23:04:44.270 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.48229.log: 73.8% -- replaced with /var/log/ceph/cluster1-client.mirror.48229.log.gz 2024-04-24T23:04:44.270 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80556.log 2024-04-24T23:04:44.271 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80556.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80556.log.gz 2024-04-24T23:04:44.271 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56323.log 2024-04-24T23:04:44.272 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56323.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56323.log.gz 2024-04-24T23:04:44.272 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81451.log 2024-04-24T23:04:44.273 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.81451.log: 90.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81451.log.gz 2024-04-24T23:04:44.273 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50747.log 2024-04-24T23:04:44.274 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.50747.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61709.log.gz 2024-04-24T23:04:44.274 INFO:teuthology.orchestra.run.smithi134.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.50747.log.gz 2024-04-24T23:04:44.274 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69805.log 2024-04-24T23:04:44.274 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63768.log 2024-04-24T23:04:44.275 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.69805.log: /var/log/ceph/cluster1-client.mirror.63768.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63768.log.gz 2024-04-24T23:04:44.275 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57195.log 2024-04-24T23:04:44.276 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57195.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.69805.log.gz 2024-04-24T23:04:44.276 INFO:teuthology.orchestra.run.smithi134.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57195.log.gz 2024-04-24T23:04:44.277 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55953.log 2024-04-24T23:04:44.277 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52684.log 2024-04-24T23:04:44.280 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.55953.log: /var/log/ceph/cluster1-client.mirror.52684.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55953.log.gz 2024-04-24T23:04:44.283 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52684.log.gz 2024-04-24T23:04:44.283 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58665.log 2024-04-24T23:04:44.283 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79273.log 2024-04-24T23:04:44.284 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58665.log: /var/log/ceph/cluster1-client.mirror.79273.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.79273.log.gz 2024-04-24T23:04:44.284 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59786.log 2024-04-24T23:04:44.284 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59786.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59786.log.gz 2024-04-24T23:04:44.284 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82318.log 2024-04-24T23:04:44.286 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82318.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.82318.log.gz 2024-04-24T23:04:44.286 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58988.log 2024-04-24T23:04:44.287 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58988.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58988.log.gz 2024-04-24T23:04:44.287 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68978.log 2024-04-24T23:04:44.288 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.68978.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.68978.log.gz 2024-04-24T23:04:44.288 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75959.log 2024-04-24T23:04:44.289 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75959.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75959.log.gz 2024-04-24T23:04:44.289 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58334.log 2024-04-24T23:04:44.290 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58334.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58334.log.gz 2024-04-24T23:04:44.290 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82670.log 2024-04-24T23:04:44.291 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.82670.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.82670.log.gz 2024-04-24T23:04:44.291 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83418.log 2024-04-24T23:04:44.292 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.83418.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.83418.log.gz 2024-04-24T23:04:44.293 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84412.log 2024-04-24T23:04:44.295 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84412.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84412.log.gz 2024-04-24T23:04:44.295 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76215.log 2024-04-24T23:04:44.296 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76215.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.76215.log.gz 2024-04-24T23:04:44.296 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60530.log 2024-04-24T23:04:44.300 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58850.log.gz 2024-04-24T23:04:44.312 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61809.log 2024-04-24T23:04:44.312 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60530.log: /var/log/ceph/cluster2-client.mirror.61809.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61809.log.gz 2024-04-24T23:04:44.312 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77258.log 2024-04-24T23:04:44.312 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77258.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.77258.log.gz 2024-04-24T23:04:44.312 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63022.log 2024-04-24T23:04:44.312 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63022.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63022.log.gz 2024-04-24T23:04:44.312 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69589.log 2024-04-24T23:04:44.312 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69589.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69589.log.gz 2024-04-24T23:04:44.312 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.77875.log 2024-04-24T23:04:44.312 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.1.77875.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.77875.log.gz 2024-04-24T23:04:44.312 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58317.log 2024-04-24T23:04:44.312 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58317.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58317.log.gz 2024-04-24T23:04:44.313 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65299.log 2024-04-24T23:04:44.313 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65299.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65299.log.gz 2024-04-24T23:04:44.313 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83483.log 2024-04-24T23:04:44.313 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83483.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.83483.log.gz 2024-04-24T23:04:44.318 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61650.log 2024-04-24T23:04:44.318 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61650.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61650.log.gz 2024-04-24T23:04:44.318 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78466.log 2024-04-24T23:04:44.318 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.78466.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.78466.log.gz 2024-04-24T23:04:44.318 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63615.log 2024-04-24T23:04:44.318 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63615.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63615.log.gz 2024-04-24T23:04:44.318 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52749.log 2024-04-24T23:04:44.318 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52749.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52749.log.gz 2024-04-24T23:04:44.318 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68434.log 2024-04-24T23:04:44.320 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68434.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68434.log.gz 2024-04-24T23:04:44.320 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53218.log 2024-04-24T23:04:44.320 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.53218.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.53218.log.gz 2024-04-24T23:04:44.321 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68740.log 2024-04-24T23:04:44.321 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.68740.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.68740.log.gz 2024-04-24T23:04:44.321 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60937.log 2024-04-24T23:04:44.321 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60937.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60937.log.gz 2024-04-24T23:04:44.321 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72862.log 2024-04-24T23:04:44.324 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.72862.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.72862.log.gz 2024-04-24T23:04:44.324 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59990.log 2024-04-24T23:04:44.325 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59990.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59990.log.gz 2024-04-24T23:04:44.325 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52042.log 2024-04-24T23:04:44.331 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52042.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67019.log 2024-04-24T23:04:44.331 INFO:teuthology.orchestra.run.smithi134.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52042.log.gz 2024-04-24T23:04:44.331 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68216.log 2024-04-24T23:04:44.332 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.67019.log: /var/log/ceph/cluster2-client.mirror.68216.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68216.log.gz 2024-04-24T23:04:44.332 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65682.log 2024-04-24T23:04:44.333 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.65682.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.67019.log.gz 2024-04-24T23:04:44.333 INFO:teuthology.orchestra.run.smithi134.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65682.log.gz 2024-04-24T23:04:44.333 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69567.log 2024-04-24T23:04:44.335 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.69567.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.69567.log.gz 2024-04-24T23:04:44.336 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61767.log 2024-04-24T23:04:44.337 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61767.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61767.log.gz 2024-04-24T23:04:44.337 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72773.log 2024-04-24T23:04:44.338 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.72773.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.72773.log.gz 2024-04-24T23:04:44.338 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85605.log 2024-04-24T23:04:44.338 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.85605.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85605.log.gz 2024-04-24T23:04:44.338 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.59379.log 2024-04-24T23:04:44.342 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.0.59379.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72701.log 2024-04-24T23:04:44.344 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72701.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72701.log.gz 2024-04-24T23:04:44.344 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77832.log 2024-04-24T23:04:44.344 INFO:teuthology.orchestra.run.smithi134.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.59379.log.gz 2024-04-24T23:04:44.346 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77832.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77832.log.gz 2024-04-24T23:04:44.347 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53727.log 2024-04-24T23:04:44.351 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53727.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47416.log 2024-04-24T23:04:44.351 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.47416.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.47416.log.gz 2024-04-24T23:04:44.352 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53727.log.gz 2024-04-24T23:04:44.352 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59235.log 2024-04-24T23:04:44.353 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59235.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59235.log.gz 2024-04-24T23:04:44.353 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66325.log 2024-04-24T23:04:44.355 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66325.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66325.log.gz 2024-04-24T23:04:44.355 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66690.log 2024-04-24T23:04:44.356 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.66690.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66690.log.gz 2024-04-24T23:04:44.356 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57748.log 2024-04-24T23:04:44.360 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57748.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80016.log 2024-04-24T23:04:44.368 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80016.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80016.log.gz 2024-04-24T23:04:44.371 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81801.log 2024-04-24T23:04:44.372 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81801.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81801.log.gz 2024-04-24T23:04:44.379 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58376.log.gz 2024-04-24T23:04:44.382 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53083.log 2024-04-24T23:04:44.388 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.53083.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53083.log.gz 2024-04-24T23:04:44.388 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72463.log 2024-04-24T23:04:44.389 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72463.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72463.log.gz 2024-04-24T23:04:44.389 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82929.log 2024-04-24T23:04:44.390 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.82929.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.82929.log.gz 2024-04-24T23:04:44.390 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73297.log 2024-04-24T23:04:44.391 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.73297.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73297.log.gz 2024-04-24T23:04:44.391 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82403.log 2024-04-24T23:04:44.392 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.82403.log: 85.7% -- replaced with /var/log/ceph/cluster1-client.mirror.82403.log.gz 2024-04-24T23:04:44.392 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81143.log 2024-04-24T23:04:44.394 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81143.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81143.log.gz 2024-04-24T23:04:44.394 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62288.log 2024-04-24T23:04:44.396 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62288.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62288.log.gz 2024-04-24T23:04:44.396 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75463.log 2024-04-24T23:04:44.397 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75463.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75463.log.gz 2024-04-24T23:04:44.397 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72313.log 2024-04-24T23:04:44.402 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.72313.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.72313.log.gz 2024-04-24T23:04:44.403 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66214.log 2024-04-24T23:04:44.403 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.66214.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66214.log.gz 2024-04-24T23:04:44.404 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64991.log 2024-04-24T23:04:44.405 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64991.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64991.log.gz 2024-04-24T23:04:44.405 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74700.log 2024-04-24T23:04:44.407 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74700.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74700.log.gz 2024-04-24T23:04:44.407 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78110.log 2024-04-24T23:04:44.408 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78110.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78110.log.gz 2024-04-24T23:04:44.408 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56563.log 2024-04-24T23:04:44.409 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79026.log 2024-04-24T23:04:44.409 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56563.log: /var/log/ceph/cluster1-client.mirror.79026.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79026.log.gz 2024-04-24T23:04:44.410 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47171.log 2024-04-24T23:04:44.410 INFO:teuthology.orchestra.run.smithi134.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56563.log.gz 2024-04-24T23:04:44.410 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.47171.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.47171.log.gz 2024-04-24T23:04:44.410 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77576.log 2024-04-24T23:04:44.411 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77576.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77576.log.gz 2024-04-24T23:04:44.411 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68788.log 2024-04-24T23:04:44.412 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59257.log.gz 2024-04-24T23:04:44.413 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.68788.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68788.log.gz 2024-04-24T23:04:44.413 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51901.log 2024-04-24T23:04:44.413 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73963.log 2024-04-24T23:04:44.413 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51901.log: /var/log/ceph/cluster2-client.mirror.73963.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51901.log.gz 2024-04-24T23:04:44.414 INFO:teuthology.orchestra.run.smithi134.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73963.log.gz 2024-04-24T23:04:44.414 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74342.log 2024-04-24T23:04:44.415 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76091.log 2024-04-24T23:04:44.416 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74342.log: /var/log/ceph/cluster2-client.mirror.76091.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76091.log.gz 2024-04-24T23:04:44.416 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64030.log 2024-04-24T23:04:44.416 INFO:teuthology.orchestra.run.smithi134.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74342.log.gz 2024-04-24T23:04:44.416 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.51354.log 2024-04-24T23:04:44.416 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.64030.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64030.log.gz 2024-04-24T23:04:44.426 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.1.51354.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65254.log 2024-04-24T23:04:44.426 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54960.log 2024-04-24T23:04:44.427 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.65254.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65254.log.gz 2024-04-24T23:04:44.427 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75731.log 2024-04-24T23:04:44.427 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54960.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54960.log.gz 2024-04-24T23:04:44.427 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84884.log 2024-04-24T23:04:44.429 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75731.log: /var/log/ceph/cluster2-client.mirror.84884.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.84884.log.gz 2024-04-24T23:04:44.429 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51738.log 2024-04-24T23:04:44.430 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51738.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75731.log.gz 2024-04-24T23:04:44.430 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74104.log 2024-04-24T23:04:44.433 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74104.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74104.log.gz 2024-04-24T23:04:44.433 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50629.log 2024-04-24T23:04:44.439 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50629.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50629.log.gz 2024-04-24T23:04:44.439 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81667.log 2024-04-24T23:04:44.440 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81667.log: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.81667.log.gz 2024-04-24T23:04:44.440 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.48067.log 2024-04-24T23:04:44.441 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.1.48067.log: 78.4% -- replaced with /var/log/ceph/cluster1-client.mirror.1.48067.log.gz 2024-04-24T23:04:44.441 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63284.log 2024-04-24T23:04:44.442 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.63284.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63284.log.gz 2024-04-24T23:04:44.442 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51651.log 2024-04-24T23:04:44.448 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.51651.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51651.log.gz 2024-04-24T23:04:44.448 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72116.log 2024-04-24T23:04:44.451 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.72116.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.72116.log.gz 2024-04-24T23:04:44.451 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62856.log 2024-04-24T23:04:44.452 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.62856.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62856.log.gz 2024-04-24T23:04:44.452 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80286.log 2024-04-24T23:04:44.464 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80286.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80286.log.gz 2024-04-24T23:04:44.464 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63419.log 2024-04-24T23:04:44.466 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63419.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63419.log.gz 2024-04-24T23:04:44.466 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53842.log 2024-04-24T23:04:44.467 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.53842.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.53842.log.gz 2024-04-24T23:04:44.467 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82835.log 2024-04-24T23:04:44.470 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82835.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57398.log.gz 2024-04-24T23:04:44.470 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71979.log 2024-04-24T23:04:44.490 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.71979.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71979.log.gz 2024-04-24T23:04:44.490 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79791.log 2024-04-24T23:04:44.502 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79791.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.79791.log.gz 2024-04-24T23:04:44.502 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85107.log 2024-04-24T23:04:44.503 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.85107.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.85107.log.gz 2024-04-24T23:04:44.503 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68454.log 2024-04-24T23:04:44.504 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68454.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68454.log.gz 2024-04-24T23:04:44.504 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75773.log 2024-04-24T23:04:44.505 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75773.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75773.log.gz 2024-04-24T23:04:44.505 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78778.log 2024-04-24T23:04:44.506 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.78778.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.78778.log.gz 2024-04-24T23:04:44.507 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64574.log 2024-04-24T23:04:44.508 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64574.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64574.log.gz 2024-04-24T23:04:44.508 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52283.log 2024-04-24T23:04:44.509 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.52283.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.52283.log.gz 2024-04-24T23:04:44.509 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55718.log 2024-04-24T23:04:44.516 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55718.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55718.log.gz 2024-04-24T23:04:44.516 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57098.log 2024-04-24T23:04:44.517 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57098.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57098.log.gz 2024-04-24T23:04:44.517 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75277.log 2024-04-24T23:04:44.518 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75277.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75277.log.gz 2024-04-24T23:04:44.518 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66303.log 2024-04-24T23:04:44.521 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.66303.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.66303.log.gz 2024-04-24T23:04:44.521 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69168.log 2024-04-24T23:04:44.530 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69168.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69168.log.gz 2024-04-24T23:04:44.530 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73362.log 2024-04-24T23:04:44.532 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.73362.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73362.log.gz 2024-04-24T23:04:44.532 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78275.log 2024-04-24T23:04:44.534 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78275.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.78275.log.gz 2024-04-24T23:04:44.534 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70200.log 2024-04-24T23:04:44.535 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.70200.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70200.log.gz 2024-04-24T23:04:44.535 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72725.log 2024-04-24T23:04:44.536 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.72725.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.72725.log.gz 2024-04-24T23:04:44.536 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75339.log 2024-04-24T23:04:44.538 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75339.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75339.log.gz 2024-04-24T23:04:44.538 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56754.log 2024-04-24T23:04:44.539 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.56754.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56754.log.gz 2024-04-24T23:04:44.539 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83654.log 2024-04-24T23:04:44.541 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83654.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.83654.log.gz 2024-04-24T23:04:44.541 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79678.log 2024-04-24T23:04:44.542 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79678.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.79678.log.gz 2024-04-24T23:04:44.542 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.53502.log 2024-04-24T23:04:44.547 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.0.53502.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.0.53502.log.gz 2024-04-24T23:04:44.547 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70652.log 2024-04-24T23:04:44.548 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.70652.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70652.log.gz 2024-04-24T23:04:44.548 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50891.log 2024-04-24T23:04:44.549 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.50891.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50891.log.gz 2024-04-24T23:04:44.549 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67382.log 2024-04-24T23:04:44.551 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.67382.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.67382.log.gz 2024-04-24T23:04:44.551 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58058.log 2024-04-24T23:04:44.552 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58058.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58058.log.gz 2024-04-24T23:04:44.552 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84173.log 2024-04-24T23:04:44.553 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84173.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.84173.log.gz 2024-04-24T23:04:44.553 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69047.log 2024-04-24T23:04:44.555 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69047.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69047.log.gz 2024-04-24T23:04:44.555 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81780.log 2024-04-24T23:04:44.556 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.81780.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.81780.log.gz 2024-04-24T23:04:44.556 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57435.log 2024-04-24T23:04:44.558 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57435.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57435.log.gz 2024-04-24T23:04:44.558 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66065.log 2024-04-24T23:04:44.560 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.66065.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.66065.log.gz 2024-04-24T23:04:44.561 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62618.log 2024-04-24T23:04:44.562 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.62618.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.62618.log.gz 2024-04-24T23:04:44.562 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53684.log 2024-04-24T23:04:44.568 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53684.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53684.log.gz 2024-04-24T23:04:44.568 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79048.log 2024-04-24T23:04:44.569 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79048.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.79048.log.gz 2024-04-24T23:04:44.569 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56603.log 2024-04-24T23:04:44.571 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56603.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56603.log.gz 2024-04-24T23:04:44.571 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63657.log 2024-04-24T23:04:44.572 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63657.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63657.log.gz 2024-04-24T23:04:44.572 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51311.log 2024-04-24T23:04:44.575 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51311.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51311.log.gz 2024-04-24T23:04:44.575 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73083.log 2024-04-24T23:04:44.576 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.73083.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.73083.log.gz 2024-04-24T23:04:44.576 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85801.log 2024-04-24T23:04:44.577 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.85801.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85801.log.gz 2024-04-24T23:04:44.577 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64812.log 2024-04-24T23:04:44.579 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64812.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64812.log.gz 2024-04-24T23:04:44.579 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76678.log 2024-04-24T23:04:44.580 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76678.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76678.log.gz 2024-04-24T23:04:44.580 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50693.log 2024-04-24T23:04:44.588 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50693.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55581.log 2024-04-24T23:04:44.608 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55581.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55581.log.gz 2024-04-24T23:04:44.608 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58791.log 2024-04-24T23:04:44.619 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58791.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58791.log.gz 2024-04-24T23:04:44.619 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84492.log 2024-04-24T23:04:44.631 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84492.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84982.log 2024-04-24T23:04:44.642 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.84982.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84982.log.gz 2024-04-24T23:04:44.642 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77278.log 2024-04-24T23:04:44.654 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77278.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77617.log 2024-04-24T23:04:44.665 INFO:teuthology.orchestra.run.smithi134.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77278.log.gz 2024-04-24T23:04:44.668 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77617.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61038.log.gz 2024-04-24T23:04:44.669 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55323.log 2024-04-24T23:04:44.673 INFO:teuthology.orchestra.run.smithi134.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.77617.log.gz 2024-04-24T23:04:44.673 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55323.log: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.51354.log.gz 2024-04-24T23:04:44.673 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58930.log 2024-04-24T23:04:44.681 INFO:teuthology.orchestra.run.smithi134.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55323.log.gz 2024-04-24T23:04:44.681 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58930.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57748.log.gz 2024-04-24T23:04:44.681 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64473.log 2024-04-24T23:04:44.688 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64473.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64473.log.gz 2024-04-24T23:04:44.688 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62404.log 2024-04-24T23:04:44.688 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.62404.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62404.log.gz 2024-04-24T23:04:44.688 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84232.log 2024-04-24T23:04:44.710 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84232.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78213.log 2024-04-24T23:04:44.721 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78213.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51008.log 2024-04-24T23:04:44.721 INFO:teuthology.orchestra.run.smithi134.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.78213.log.gz 2024-04-24T23:04:44.754 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51008.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83462.log 2024-04-24T23:04:44.754 INFO:teuthology.orchestra.run.smithi134.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51008.log.gz 2024-04-24T23:04:44.765 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83462.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.83462.log.gz 2024-04-24T23:04:44.765 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61505.log 2024-04-24T23:04:44.765 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60530.log.gz 2024-04-24T23:04:44.765 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61505.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61505.log.gz 2024-04-24T23:04:44.765 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72963.log 2024-04-24T23:04:44.765 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72963.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72963.log.gz 2024-04-24T23:04:44.765 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46927.log 2024-04-24T23:04:44.765 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.46927.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.46927.log.gz 2024-04-24T23:04:44.765 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63881.log 2024-04-24T23:04:44.776 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.63881.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63881.log.gz 2024-04-24T23:04:44.776 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80331.log 2024-04-24T23:04:44.776 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80331.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.80331.log.gz 2024-04-24T23:04:44.776 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48666.log 2024-04-24T23:04:44.776 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.48666.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.48666.log.gz 2024-04-24T23:04:44.776 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58354.log 2024-04-24T23:04:44.776 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58354.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58354.log.gz 2024-04-24T23:04:44.776 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81842.log 2024-04-24T23:04:44.776 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81842.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81842.log.gz 2024-04-24T23:04:44.776 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82973.log 2024-04-24T23:04:44.776 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82973.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82973.log.gz 2024-04-24T23:04:44.776 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74720.log 2024-04-24T23:04:44.776 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.74720.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74720.log.gz 2024-04-24T23:04:44.776 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71437.log 2024-04-24T23:04:44.782 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71437.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71437.log.gz 2024-04-24T23:04:44.782 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73773.log 2024-04-24T23:04:44.782 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.73773.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73773.log.gz 2024-04-24T23:04:44.782 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76049.log 2024-04-24T23:04:44.782 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76049.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76049.log.gz 2024-04-24T23:04:44.782 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75607.log 2024-04-24T23:04:44.792 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75607.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75607.log.gz 2024-04-24T23:04:44.793 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77133.log 2024-04-24T23:04:44.793 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77133.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77133.log.gz 2024-04-24T23:04:44.793 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.65034.log 2024-04-24T23:04:44.793 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.0.65034.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.0.65034.log.gz 2024-04-24T23:04:44.793 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73842.log 2024-04-24T23:04:44.793 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73842.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73842.log.gz 2024-04-24T23:04:44.793 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83610.log 2024-04-24T23:04:44.793 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.83610.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.83610.log.gz 2024-04-24T23:04:44.793 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60341.log 2024-04-24T23:04:44.793 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60341.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60341.log.gz 2024-04-24T23:04:44.793 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77050.log 2024-04-24T23:04:44.793 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77050.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.77050.log.gz 2024-04-24T23:04:44.793 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84574.log 2024-04-24T23:04:44.804 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84574.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.84574.log.gz 2024-04-24T23:04:44.804 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62044.log 2024-04-24T23:04:44.804 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.62044.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62044.log.gz 2024-04-24T23:04:44.804 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78027.log 2024-04-24T23:04:44.804 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78027.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.78027.log.gz 2024-04-24T23:04:44.804 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62445.log 2024-04-24T23:04:44.804 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.62445.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62445.log.gz 2024-04-24T23:04:44.804 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77410.log 2024-04-24T23:04:44.804 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77410.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.77410.log.gz 2024-04-24T23:04:44.804 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.84729.log 2024-04-24T23:04:44.804 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.1.84729.log: 91.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.84729.log.gz 2024-04-24T23:04:44.804 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85507.log 2024-04-24T23:04:44.804 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.85507.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85507.log.gz 2024-04-24T23:04:44.815 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55601.log 2024-04-24T23:04:44.815 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.55601.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55601.log.gz 2024-04-24T23:04:44.815 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80760.log 2024-04-24T23:04:44.815 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80760.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.80760.log.gz 2024-04-24T23:04:44.815 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77237.log 2024-04-24T23:04:44.815 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77237.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.77237.log.gz 2024-04-24T23:04:44.815 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73146.log 2024-04-24T23:04:44.815 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73146.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73146.log.gz 2024-04-24T23:04:44.816 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.62116.log 2024-04-24T23:04:44.826 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.2.62116.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56523.log 2024-04-24T23:04:44.826 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.2.62116.log.gz 2024-04-24T23:04:44.826 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56523.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56523.log.gz 2024-04-24T23:04:44.826 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55386.log 2024-04-24T23:04:44.826 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.55386.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55386.log.gz 2024-04-24T23:04:44.826 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69526.log 2024-04-24T23:04:44.832 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.69526.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.69526.log.gz 2024-04-24T23:04:44.832 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48632.log 2024-04-24T23:04:44.832 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.48632.log: 67.4% -- replaced with /var/log/ceph/cluster1-client.admin.48632.log.gz 2024-04-24T23:04:44.832 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71346.log 2024-04-24T23:04:44.832 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71346.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71346.log.gz 2024-04-24T23:04:44.832 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60091.log 2024-04-24T23:04:44.843 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60091.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55867.log 2024-04-24T23:04:44.863 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.55867.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55867.log.gz 2024-04-24T23:04:44.864 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78531.log 2024-04-24T23:04:44.875 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.78531.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78531.log.gz 2024-04-24T23:04:44.875 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77340.log 2024-04-24T23:04:44.875 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58665.log.gz 2024-04-24T23:04:44.886 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77340.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.77340.log.gz 2024-04-24T23:04:44.886 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55301.log 2024-04-24T23:04:44.886 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.55301.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55301.log.gz 2024-04-24T23:04:44.886 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50963.log 2024-04-24T23:04:44.886 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.50963.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50963.log.gz 2024-04-24T23:04:44.886 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57886.log 2024-04-24T23:04:44.886 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57886.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57886.log.gz 2024-04-24T23:04:44.886 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74801.log 2024-04-24T23:04:44.886 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74801.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74801.log.gz 2024-04-24T23:04:44.886 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47759.log 2024-04-24T23:04:44.887 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.47759.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.47759.log.gz 2024-04-24T23:04:44.887 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53959.log 2024-04-24T23:04:44.887 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.53959.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53959.log.gz 2024-04-24T23:04:44.887 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69454.log 2024-04-24T23:04:44.887 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.69454.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69454.log.gz 2024-04-24T23:04:44.887 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76451.log 2024-04-24T23:04:44.897 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76451.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.76451.log.gz 2024-04-24T23:04:44.897 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74563.log 2024-04-24T23:04:44.897 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74563.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74563.log.gz 2024-04-24T23:04:44.897 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66283.log 2024-04-24T23:04:44.897 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66283.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66283.log.gz 2024-04-24T23:04:44.897 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52727.log 2024-04-24T23:04:44.897 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.52727.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52727.log.gz 2024-04-24T23:04:44.897 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80795.log 2024-04-24T23:04:44.907 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80795.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.80795.log.gz 2024-04-24T23:04:44.908 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73035.log 2024-04-24T23:04:44.908 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.73035.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73035.log.gz 2024-04-24T23:04:44.908 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50552.log 2024-04-24T23:04:44.908 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.50552.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.50552.log.gz 2024-04-24T23:04:44.908 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64711.log 2024-04-24T23:04:44.908 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64711.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64711.log.gz 2024-04-24T23:04:44.908 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58712.log 2024-04-24T23:04:44.908 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58712.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58712.log.gz 2024-04-24T23:04:44.908 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66087.log 2024-04-24T23:04:44.908 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66087.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66087.log.gz 2024-04-24T23:04:44.908 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81223.log 2024-04-24T23:04:44.919 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81223.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83905.log 2024-04-24T23:04:44.930 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.83905.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.83905.log.gz 2024-04-24T23:04:44.940 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71287.log 2024-04-24T23:04:44.942 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.82835.log.gz 2024-04-24T23:04:44.944 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71287.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71287.log.gz 2024-04-24T23:04:44.944 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64428.log 2024-04-24T23:04:44.945 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.64428.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64428.log.gz 2024-04-24T23:04:44.945 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83524.log 2024-04-24T23:04:44.946 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83524.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.83524.log.gz 2024-04-24T23:04:44.946 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56800.log 2024-04-24T23:04:44.947 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.56800.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56800.log.gz 2024-04-24T23:04:44.947 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62332.log 2024-04-24T23:04:44.948 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.62332.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62332.log.gz 2024-04-24T23:04:44.948 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76533.log 2024-04-24T23:04:44.951 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76533.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76533.log.gz 2024-04-24T23:04:44.952 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69285.log 2024-04-24T23:04:44.954 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69285.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69285.log.gz 2024-04-24T23:04:44.954 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66738.log 2024-04-24T23:04:44.955 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.66738.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66738.log.gz 2024-04-24T23:04:44.955 INFO:teuthology.orchestra.run.smithi134.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51738.log.gz 2024-04-24T23:04:44.955 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47808.log 2024-04-24T23:04:44.955 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57906.log 2024-04-24T23:04:44.956 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.47808.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.47808.log.gz 2024-04-24T23:04:44.956 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50413.log 2024-04-24T23:04:44.956 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57906.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57906.log.gz 2024-04-24T23:04:44.957 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66476.log 2024-04-24T23:04:44.957 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.66476.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50413.log.gz 2024-04-24T23:04:44.958 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67732.log 2024-04-24T23:04:44.958 INFO:teuthology.orchestra.run.smithi134.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66476.log.gz 2024-04-24T23:04:44.958 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57157.log 2024-04-24T23:04:44.958 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.67732.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.67732.log.gz 2024-04-24T23:04:44.959 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57157.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84530.log 2024-04-24T23:04:44.960 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.84530.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.84530.log.gz 2024-04-24T23:04:44.960 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.48229.log 2024-04-24T23:04:44.962 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.1.48229.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.1.48229.log.gz 2024-04-24T23:04:44.962 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85007.log 2024-04-24T23:04:44.963 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.85007.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85007.log.gz 2024-04-24T23:04:44.963 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66642.log 2024-04-24T23:04:44.964 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66642.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66642.log.gz 2024-04-24T23:04:44.965 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75008.log 2024-04-24T23:04:44.966 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75008.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75008.log.gz 2024-04-24T23:04:44.966 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77770.log 2024-04-24T23:04:44.969 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77770.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77770.log.gz 2024-04-24T23:04:44.969 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61400.log 2024-04-24T23:04:44.971 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61400.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61400.log.gz 2024-04-24T23:04:44.971 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51167.log 2024-04-24T23:04:44.971 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51167.log: 73.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51167.log.gz 2024-04-24T23:04:44.971 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51538.log 2024-04-24T23:04:44.973 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.51538.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51538.log.gz 2024-04-24T23:04:44.973 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50649.log 2024-04-24T23:04:44.979 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.50649.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50649.log.gz 2024-04-24T23:04:44.979 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62594.log 2024-04-24T23:04:44.980 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.62594.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62594.log.gz 2024-04-24T23:04:44.980 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59155.log 2024-04-24T23:04:44.981 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59155.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59155.log.gz 2024-04-24T23:04:44.981 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74321.log 2024-04-24T23:04:44.982 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.74321.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.74321.log.gz 2024-04-24T23:04:44.983 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.50205.log 2024-04-24T23:04:44.986 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.1.50205.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.1.50205.log.gz 2024-04-24T23:04:44.986 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84327.log 2024-04-24T23:04:44.987 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.84327.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84327.log.gz 2024-04-24T23:04:44.987 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82693.log 2024-04-24T23:04:44.988 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.82693.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.82693.log.gz 2024-04-24T23:04:44.988 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74742.log 2024-04-24T23:04:44.990 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74742.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74742.log.gz 2024-04-24T23:04:44.990 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78130.log 2024-04-24T23:04:44.993 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78130.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.78130.log.gz 2024-04-24T23:04:44.993 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71128.log 2024-04-24T23:04:44.994 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71128.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71128.log.gz 2024-04-24T23:04:44.994 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71245.log 2024-04-24T23:04:44.996 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71245.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71245.log.gz 2024-04-24T23:04:44.997 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57668.log 2024-04-24T23:04:45.002 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57668.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61459.log 2024-04-24T23:04:45.004 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58930.log.gz 2024-04-24T23:04:45.018 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61459.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54587.log 2024-04-24T23:04:45.035 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54587.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54587.log.gz 2024-04-24T23:04:45.045 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51694.log 2024-04-24T23:04:45.057 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.51694.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51694.log.gz 2024-04-24T23:04:45.068 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79408.log 2024-04-24T23:04:45.079 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79408.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79408.log.gz 2024-04-24T23:04:45.079 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77153.log 2024-04-24T23:04:45.090 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77153.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59089.log 2024-04-24T23:04:45.090 INFO:teuthology.orchestra.run.smithi134.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.77153.log.gz 2024-04-24T23:04:45.103 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59089.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68042.log 2024-04-24T23:04:45.115 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.68042.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68042.log.gz 2024-04-24T23:04:45.115 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70390.log 2024-04-24T23:04:45.139 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70390.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70390.log.gz 2024-04-24T23:04:45.139 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83072.log 2024-04-24T23:04:45.149 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83072.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.83072.log.gz 2024-04-24T23:04:45.150 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57848.log 2024-04-24T23:04:45.172 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57848.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82360.log 2024-04-24T23:04:45.183 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82360.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71955.log 2024-04-24T23:04:45.186 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.84492.log.gz 2024-04-24T23:04:45.187 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71955.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71955.log.gz 2024-04-24T23:04:45.187 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66166.log 2024-04-24T23:04:45.188 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66166.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66166.log.gz 2024-04-24T23:04:45.189 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76947.log 2024-04-24T23:04:45.190 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76947.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76947.log.gz 2024-04-24T23:04:45.190 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48683.log 2024-04-24T23:04:45.191 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.48683.log: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.48683.log.gz 2024-04-24T23:04:45.191 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47857.log 2024-04-24T23:04:45.192 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.47857.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.47857.log.gz 2024-04-24T23:04:45.192 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81102.log 2024-04-24T23:04:45.193 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81102.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.81102.log.gz 2024-04-24T23:04:45.193 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83753.log 2024-04-24T23:04:45.194 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83753.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.83753.log.gz 2024-04-24T23:04:45.194 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61437.log 2024-04-24T23:04:45.196 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61437.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61437.log.gz 2024-04-24T23:04:45.196 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83794.log 2024-04-24T23:04:45.197 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83794.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83794.log.gz 2024-04-24T23:04:45.197 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69547.log 2024-04-24T23:04:45.199 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69547.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69547.log.gz 2024-04-24T23:04:45.199 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59177.log 2024-04-24T23:04:45.207 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59177.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70224.log 2024-04-24T23:04:45.220 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.70224.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70224.log.gz 2024-04-24T23:04:45.220 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72075.log 2024-04-24T23:04:45.242 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.72075.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.72075.log.gz 2024-04-24T23:04:45.242 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59970.log 2024-04-24T23:04:45.248 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59970.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59970.log.gz 2024-04-24T23:04:45.248 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58206.log 2024-04-24T23:04:45.271 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58206.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78337.log 2024-04-24T23:04:45.285 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78337.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.78337.log.gz 2024-04-24T23:04:45.285 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64949.log 2024-04-24T23:04:45.300 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64949.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64949.log.gz 2024-04-24T23:04:45.300 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79656.log 2024-04-24T23:04:45.317 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57157.log.gz 2024-04-24T23:04:45.327 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79656.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79656.log.gz 2024-04-24T23:04:45.327 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69827.log 2024-04-24T23:04:45.338 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.81223.log.gz 2024-04-24T23:04:45.338 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69827.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72600.log 2024-04-24T23:04:45.338 INFO:teuthology.orchestra.run.smithi134.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69827.log.gz 2024-04-24T23:04:45.362 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.84232.log.gz 2024-04-24T23:04:45.373 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.72600.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72600.log.gz 2024-04-24T23:04:45.373 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62425.log 2024-04-24T23:04:45.391 INFO:teuthology.orchestra.run.smithi134.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50693.log.gz 2024-04-24T23:04:45.394 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62425.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62425.log.gz 2024-04-24T23:04:45.394 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82994.log 2024-04-24T23:04:45.395 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82994.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82994.log.gz 2024-04-24T23:04:45.395 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54076.log 2024-04-24T23:04:45.402 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54076.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54076.log.gz 2024-04-24T23:04:45.402 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67192.log 2024-04-24T23:04:45.403 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.67192.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.67192.log.gz 2024-04-24T23:04:45.403 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60150.log 2024-04-24T23:04:45.404 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60150.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60150.log.gz 2024-04-24T23:04:45.404 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51925.log 2024-04-24T23:04:45.405 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.51925.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.51925.log.gz 2024-04-24T23:04:45.405 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82776.log 2024-04-24T23:04:45.406 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82776.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.82776.log.gz 2024-04-24T23:04:45.406 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80488.log 2024-04-24T23:04:45.408 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80488.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80488.log.gz 2024-04-24T23:04:45.408 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50160.log 2024-04-24T23:04:45.409 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.50160.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.50160.log.gz 2024-04-24T23:04:45.409 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51120.log 2024-04-24T23:04:45.417 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51120.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57668.log.gz 2024-04-24T23:04:45.417 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76132.log 2024-04-24T23:04:45.443 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76132.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74035.log 2024-04-24T23:04:45.443 INFO:teuthology.orchestra.run.smithi134.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76132.log.gz 2024-04-24T23:04:45.455 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.74035.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74035.log.gz 2024-04-24T23:04:45.455 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53148.log 2024-04-24T23:04:45.469 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53148.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55253.log 2024-04-24T23:04:45.480 INFO:teuthology.orchestra.run.smithi134.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53148.log.gz 2024-04-24T23:04:45.488 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55253.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65420.log 2024-04-24T23:04:45.489 INFO:teuthology.orchestra.run.smithi134.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55253.log.gz 2024-04-24T23:04:45.489 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60091.log.gz 2024-04-24T23:04:45.489 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65420.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65420.log.gz 2024-04-24T23:04:45.489 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59597.log 2024-04-24T23:04:45.489 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59597.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59597.log.gz 2024-04-24T23:04:45.489 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57032.log 2024-04-24T23:04:45.489 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57032.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57032.log.gz 2024-04-24T23:04:45.489 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77875.log 2024-04-24T23:04:45.489 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.77875.log: 86.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77875.log.gz 2024-04-24T23:04:45.489 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55233.log 2024-04-24T23:04:45.489 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55233.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.55233.log.gz 2024-04-24T23:04:45.489 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55032.log 2024-04-24T23:04:45.489 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59089.log.gz 2024-04-24T23:04:45.489 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.55032.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55032.log.gz 2024-04-24T23:04:45.490 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75938.log 2024-04-24T23:04:45.490 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75938.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75938.log.gz 2024-04-24T23:04:45.490 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70438.log 2024-04-24T23:04:45.491 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.70438.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70438.log.gz 2024-04-24T23:04:45.491 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77216.log 2024-04-24T23:04:45.494 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77216.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.77216.log.gz 2024-04-24T23:04:45.494 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61571.log 2024-04-24T23:04:45.494 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61571.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61571.log.gz 2024-04-24T23:04:45.494 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75979.log 2024-04-24T23:04:45.497 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75979.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75979.log.gz 2024-04-24T23:04:45.497 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71853.log 2024-04-24T23:04:45.500 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.71853.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71853.log.gz 2024-04-24T23:04:45.500 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57178.log 2024-04-24T23:04:45.500 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57178.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57178.log.gz 2024-04-24T23:04:45.500 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55538.log 2024-04-24T23:04:45.502 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55538.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61459.log.gz 2024-04-24T23:04:45.506 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71027.log 2024-04-24T23:04:45.506 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.71027.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55538.log.gz 2024-04-24T23:04:45.508 INFO:teuthology.orchestra.run.smithi134.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71027.log.gz 2024-04-24T23:04:45.508 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72443.log 2024-04-24T23:04:45.508 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76029.log 2024-04-24T23:04:45.509 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72443.log: /var/log/ceph/cluster2-client.mirror.76029.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76029.log.gz 2024-04-24T23:04:45.509 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81037.log 2024-04-24T23:04:45.509 INFO:teuthology.orchestra.run.smithi134.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72443.log.gz 2024-04-24T23:04:45.510 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.51167.log 2024-04-24T23:04:45.511 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81037.log: /var/log/ceph/cluster1-client.mirror.2.51167.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.81037.log.gz 2024-04-24T23:04:45.512 INFO:teuthology.orchestra.run.smithi134.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.2.51167.log.gz 2024-04-24T23:04:45.512 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79768.log 2024-04-24T23:04:45.513 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48230.log 2024-04-24T23:04:45.513 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79768.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.79768.log.gz 2024-04-24T23:04:45.514 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.48230.log: 81.5% -- replaced with /var/log/ceph/cluster1-client.mirror.48230.log.gz 2024-04-24T23:04:45.514 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66541.log 2024-04-24T23:04:45.516 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.66541.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.66541.log.gz 2024-04-24T23:04:45.516 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75194.log 2024-04-24T23:04:45.516 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53868.log 2024-04-24T23:04:45.517 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75194.log: /var/log/ceph/cluster2-client.mirror.53868.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75194.log.gz 2024-04-24T23:04:45.518 INFO:teuthology.orchestra.run.smithi134.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53868.log.gz 2024-04-24T23:04:45.518 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66978.log 2024-04-24T23:04:45.519 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76885.log 2024-04-24T23:04:45.519 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.66978.log: /var/log/ceph/cluster2-client.mirror.76885.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.66978.log.gz 2024-04-24T23:04:45.519 INFO:teuthology.orchestra.run.smithi134.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.76885.log.gz 2024-04-24T23:04:45.520 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77660.log 2024-04-24T23:04:45.520 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56818.log 2024-04-24T23:04:45.521 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77660.log: /var/log/ceph/cluster2-client.mirror.56818.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56818.log.gz 2024-04-24T23:04:45.521 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72884.log 2024-04-24T23:04:45.527 INFO:teuthology.orchestra.run.smithi134.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77660.log.gz 2024-04-24T23:04:45.527 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72884.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72884.log.gz 2024-04-24T23:04:45.527 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55996.log 2024-04-24T23:04:45.527 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59697.log 2024-04-24T23:04:45.531 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.55996.log: /var/log/ceph/cluster2-client.mirror.59697.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59697.log.gz 2024-04-24T23:04:45.531 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75173.log 2024-04-24T23:04:45.531 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75173.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75173.log.gz 2024-04-24T23:04:45.531 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59580.log 2024-04-24T23:04:45.531 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59580.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59580.log.gz 2024-04-24T23:04:45.531 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73749.log 2024-04-24T23:04:45.532 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.73749.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55996.log.gz 2024-04-24T23:04:45.532 INFO:teuthology.orchestra.run.smithi134.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73749.log.gz 2024-04-24T23:04:45.532 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84090.log 2024-04-24T23:04:45.532 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68107.log 2024-04-24T23:04:45.532 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.84090.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.84090.log.gz 2024-04-24T23:04:45.533 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.68107.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68107.log.gz 2024-04-24T23:04:45.534 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-04-24T23:04:45.534 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69216.log 2024-04-24T23:04:45.539 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2.log: /var/log/ceph/cluster1-client.mirror.69216.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69216.log.gz 2024-04-24T23:04:45.539 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67454.log 2024-04-24T23:04:45.539 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.67454.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.67454.log.gz 2024-04-24T23:04:45.539 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80128.log 2024-04-24T23:04:45.539 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80128.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.80128.log.gz 2024-04-24T23:04:45.539 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59491.log 2024-04-24T23:04:45.539 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59491.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59491.log.gz 2024-04-24T23:04:45.539 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68716.log 2024-04-24T23:04:45.539 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.68716.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.68716.log.gz 2024-04-24T23:04:45.539 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50360.log 2024-04-24T23:04:45.540 INFO:teuthology.orchestra.run.smithi134.stderr: 86.1% -- replaced with /var/log/ceph/cluster2.log.gz 2024-04-24T23:04:45.540 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.50360.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.50360.log.gz 2024-04-24T23:04:45.540 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76278.log 2024-04-24T23:04:45.540 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67338.log 2024-04-24T23:04:45.541 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.76278.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76278.log.gz 2024-04-24T23:04:45.541 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67338.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67338.log.gz 2024-04-24T23:04:45.541 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54139.log 2024-04-24T23:04:45.542 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78420.log 2024-04-24T23:04:45.547 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.54139.log: /var/log/ceph/cluster2-client.mirror.78420.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78420.log.gz 2024-04-24T23:04:45.547 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81058.log 2024-04-24T23:04:45.547 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.81058.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81058.log.gz 2024-04-24T23:04:45.548 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79723.log 2024-04-24T23:04:45.548 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79723.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54139.log.gz 2024-04-24T23:04:45.548 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.79723.log.gz 2024-04-24T23:04:45.548 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59845.log 2024-04-24T23:04:45.549 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60742.log 2024-04-24T23:04:45.549 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59845.log: /var/log/ceph/cluster2-client.mirror.60742.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60742.log.gz 2024-04-24T23:04:45.549 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54672.log 2024-04-24T23:04:45.550 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54672.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.54672.log.gz 2024-04-24T23:04:45.550 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60408.log 2024-04-24T23:04:45.551 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60408.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60408.log.gz 2024-04-24T23:04:45.551 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72241.log 2024-04-24T23:04:45.552 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.72241.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.72241.log.gz 2024-04-24T23:04:45.552 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84067.log 2024-04-24T23:04:45.553 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.84067.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84067.log.gz 2024-04-24T23:04:45.553 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83561.log 2024-04-24T23:04:45.554 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83561.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.83561.log.gz 2024-04-24T23:04:45.554 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63308.log 2024-04-24T23:04:45.555 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.63308.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63308.log.gz 2024-04-24T23:04:45.555 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69502.log 2024-04-24T23:04:45.556 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.69502.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69502.log.gz 2024-04-24T23:04:45.556 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52085.log 2024-04-24T23:04:45.562 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52085.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52085.log.gz 2024-04-24T23:04:45.562 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59030.log 2024-04-24T23:04:45.562 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81572.log 2024-04-24T23:04:45.563 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59030.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59030.log.gz 2024-04-24T23:04:45.563 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.81572.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.81572.log.gz 2024-04-24T23:04:45.564 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59803.log 2024-04-24T23:04:45.565 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59803.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59803.log.gz 2024-04-24T23:04:45.565 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68353.log 2024-04-24T23:04:45.567 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.68353.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68353.log.gz 2024-04-24T23:04:45.567 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57689.log 2024-04-24T23:04:45.568 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57689.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57689.log.gz 2024-04-24T23:04:45.568 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73384.log 2024-04-24T23:04:45.569 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73384.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73384.log.gz 2024-04-24T23:04:45.569 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77452.log 2024-04-24T23:04:45.570 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77452.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77452.log.gz 2024-04-24T23:04:45.570 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71788.log 2024-04-24T23:04:45.571 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.71788.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71788.log.gz 2024-04-24T23:04:45.571 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78296.log 2024-04-24T23:04:45.572 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78296.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.78296.log.gz 2024-04-24T23:04:45.573 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74845.log 2024-04-24T23:04:45.573 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.74845.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74845.log.gz 2024-04-24T23:04:45.573 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.54226.log 2024-04-24T23:04:45.580 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.0.54226.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.0.54226.log.gz 2024-04-24T23:04:45.580 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64880.log 2024-04-24T23:04:45.580 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.82360.log.gz 2024-04-24T23:04:45.580 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.64880.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64880.log.gz 2024-04-24T23:04:45.581 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73463.log 2024-04-24T23:04:45.581 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78823.log 2024-04-24T23:04:45.582 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73463.log: /var/log/ceph/cluster1-client.mirror.78823.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73463.log.gz 2024-04-24T23:04:45.582 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78823.log.gz 2024-04-24T23:04:45.582 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78048.log 2024-04-24T23:04:45.582 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79993.log 2024-04-24T23:04:45.583 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78048.log: /var/log/ceph/cluster1-client.mirror.79993.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.78048.log.gz 2024-04-24T23:04:45.583 INFO:teuthology.orchestra.run.smithi134.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.79993.log.gz 2024-04-24T23:04:45.583 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60172.log 2024-04-24T23:04:45.584 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66024.log 2024-04-24T23:04:45.589 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60172.log: /var/log/ceph/cluster1-client.mirror.66024.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66024.log.gz 2024-04-24T23:04:45.589 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50867.log 2024-04-24T23:04:45.589 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50867.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50867.log.gz 2024-04-24T23:04:45.589 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75256.log 2024-04-24T23:04:45.590 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75256.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75256.log.gz 2024-04-24T23:04:45.590 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79116.log 2024-04-24T23:04:45.590 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79116.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79116.log.gz 2024-04-24T23:04:45.590 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75442.log 2024-04-24T23:04:45.590 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75442.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59177.log.gz 2024-04-24T23:04:45.590 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51771.log 2024-04-24T23:04:45.591 INFO:teuthology.orchestra.run.smithi134.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75442.log.gz 2024-04-24T23:04:45.591 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.51771.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.51771.log.gz 2024-04-24T23:04:45.591 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78089.log 2024-04-24T23:04:45.591 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.56818.log 2024-04-24T23:04:45.591 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78089.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.78089.log.gz 2024-04-24T23:04:45.597 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.1.56818.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69740.log 2024-04-24T23:04:45.597 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.69740.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56995.log 2024-04-24T23:04:45.598 INFO:teuthology.orchestra.run.smithi134.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69740.log.gz 2024-04-24T23:04:45.598 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56995.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56995.log.gz 2024-04-24T23:04:45.598 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.49481.log 2024-04-24T23:04:45.604 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.1.49481.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84210.log 2024-04-24T23:04:45.604 INFO:teuthology.orchestra.run.smithi134.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.56818.log.gz 2024-04-24T23:04:45.605 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84210.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.84210.log.gz 2024-04-24T23:04:45.605 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84470.log 2024-04-24T23:04:45.606 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84470.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.84470.log.gz 2024-04-24T23:04:45.606 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70144.log 2024-04-24T23:04:45.606 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70144.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70144.log.gz 2024-04-24T23:04:45.607 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67804.log 2024-04-24T23:04:45.607 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.67804.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.67804.log.gz 2024-04-24T23:04:45.608 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63546.log 2024-04-24T23:04:45.609 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.63546.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63546.log.gz 2024-04-24T23:04:45.609 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79588.log 2024-04-24T23:04:45.609 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79588.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79588.log.gz 2024-04-24T23:04:45.609 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49270.log 2024-04-24T23:04:45.611 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49270.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49270.log.gz 2024-04-24T23:04:45.611 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54694.log 2024-04-24T23:04:45.613 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54694.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63356.log 2024-04-24T23:04:45.620 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.63356.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63356.log.gz 2024-04-24T23:04:45.620 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73249.log 2024-04-24T23:04:45.621 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.73249.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73249.log.gz 2024-04-24T23:04:45.621 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57257.log 2024-04-24T23:04:45.622 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57257.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57257.log.gz 2024-04-24T23:04:45.622 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71615.log 2024-04-24T23:04:45.625 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.71615.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71615.log.gz 2024-04-24T23:04:45.625 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48004.log 2024-04-24T23:04:45.626 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.48004.log: 74.1% -- replaced with /var/log/ceph/cluster2-client.mirror.48004.log.gz 2024-04-24T23:04:45.626 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75504.log 2024-04-24T23:04:45.627 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75504.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75504.log.gz 2024-04-24T23:04:45.627 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79251.log 2024-04-24T23:04:45.628 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79251.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79251.log.gz 2024-04-24T23:04:45.629 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69386.log 2024-04-24T23:04:45.639 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69386.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69386.log.gz 2024-04-24T23:04:45.639 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47465.log 2024-04-24T23:04:45.640 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.47465.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.47465.log.gz 2024-04-24T23:04:45.640 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79138.log 2024-04-24T23:04:45.641 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79138.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.79138.log.gz 2024-04-24T23:04:45.642 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64731.log 2024-04-24T23:04:45.644 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.64731.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64731.log.gz 2024-04-24T23:04:45.644 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47661.log 2024-04-24T23:04:45.645 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.47661.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.47661.log.gz 2024-04-24T23:04:45.645 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.52193.log 2024-04-24T23:04:45.648 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.2.52193.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.2.52193.log.gz 2024-04-24T23:04:45.648 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46929.log 2024-04-24T23:04:45.649 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.46929.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.46929.log.gz 2024-04-24T23:04:45.649 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50939.log 2024-04-24T23:04:45.650 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.50939.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50939.log.gz 2024-04-24T23:04:45.650 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67950.log 2024-04-24T23:04:45.652 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67950.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67950.log.gz 2024-04-24T23:04:45.652 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66779.log 2024-04-24T23:04:45.654 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.66779.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66779.log.gz 2024-04-24T23:04:45.655 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80241.log 2024-04-24T23:04:45.656 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80241.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.80241.log.gz 2024-04-24T23:04:45.656 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54031.log 2024-04-24T23:04:45.657 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.54031.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54031.log.gz 2024-04-24T23:04:45.657 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76409.log 2024-04-24T23:04:45.660 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76409.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76409.log.gz 2024-04-24T23:04:45.660 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59010.log 2024-04-24T23:04:45.667 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59010.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55890.log 2024-04-24T23:04:45.667 INFO:teuthology.orchestra.run.smithi134.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.57848.log.gz 2024-04-24T23:04:45.674 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55890.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55890.log.gz 2024-04-24T23:04:45.675 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60759.log 2024-04-24T23:04:45.686 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60759.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70962.log 2024-04-24T23:04:45.686 INFO:teuthology.orchestra.run.smithi134.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60759.log.gz 2024-04-24T23:04:45.698 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.70962.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70962.log.gz 2024-04-24T23:04:45.698 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77555.log 2024-04-24T23:04:45.711 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77555.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78846.log 2024-04-24T23:04:45.711 INFO:teuthology.orchestra.run.smithi134.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77555.log.gz 2024-04-24T23:04:45.712 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.78846.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.78846.log.gz 2024-04-24T23:04:45.712 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85409.log 2024-04-24T23:04:45.713 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.85409.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85409.log.gz 2024-04-24T23:04:45.713 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76781.log 2024-04-24T23:04:45.716 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76781.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76781.log.gz 2024-04-24T23:04:45.717 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71176.log 2024-04-24T23:04:45.718 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.71176.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71176.log.gz 2024-04-24T23:04:45.718 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83035.log 2024-04-24T23:04:45.719 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83035.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.83035.log.gz 2024-04-24T23:04:45.719 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84819.log 2024-04-24T23:04:45.720 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.84819.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84819.log.gz 2024-04-24T23:04:45.720 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81595.log 2024-04-24T23:04:45.721 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.81595.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.81595.log.gz 2024-04-24T23:04:45.721 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60113.log 2024-04-24T23:04:45.722 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60113.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60113.log.gz 2024-04-24T23:04:45.722 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63594.log 2024-04-24T23:04:45.724 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.63594.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63594.log.gz 2024-04-24T23:04:45.724 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63522.log 2024-04-24T23:04:45.725 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.63522.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63522.log.gz 2024-04-24T23:04:45.725 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54984.log 2024-04-24T23:04:45.726 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.54984.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54984.log.gz 2024-04-24T23:04:45.726 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48067.log 2024-04-24T23:04:45.727 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.48067.log: 49.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48067.log.gz 2024-04-24T23:04:45.727 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78358.log 2024-04-24T23:04:45.730 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78358.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.78358.log.gz 2024-04-24T23:04:45.731 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51514.log 2024-04-24T23:04:45.731 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.51514.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51514.log.gz 2024-04-24T23:04:45.731 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46632.log 2024-04-24T23:04:45.731 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.46632.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.46632.log.gz 2024-04-24T23:04:45.731 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63240.log 2024-04-24T23:04:45.732 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63240.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63240.log.gz 2024-04-24T23:04:45.733 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70170.log 2024-04-24T23:04:45.734 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.70170.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70170.log.gz 2024-04-24T23:04:45.734 INFO:teuthology.orchestra.run.smithi134.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-client.mirror.1.49481.log.gz 2024-04-24T23:04:45.734 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61846.log 2024-04-24T23:04:45.735 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61846.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61846.log.gz 2024-04-24T23:04:45.735 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76173.log 2024-04-24T23:04:45.735 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56217.log 2024-04-24T23:04:45.738 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76173.log: /var/log/ceph/cluster2-client.mirror.56217.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76173.log.gz 2024-04-24T23:04:45.741 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56217.log.gz 2024-04-24T23:04:45.741 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81081.log 2024-04-24T23:04:45.742 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.81081.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.81081.log.gz 2024-04-24T23:04:45.742 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73725.log 2024-04-24T23:04:45.743 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.73725.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73725.log.gz 2024-04-24T23:04:45.743 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74821.log 2024-04-24T23:04:45.744 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74821.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74821.log.gz 2024-04-24T23:04:45.744 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76926.log 2024-04-24T23:04:45.745 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76926.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76926.log.gz 2024-04-24T23:04:45.745 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82879.log 2024-04-24T23:04:45.746 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.82879.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.82879.log.gz 2024-04-24T23:04:45.746 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61126.log 2024-04-24T23:04:45.753 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61126.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62002.log 2024-04-24T23:04:45.754 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62002.log: 2.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62002.log.gz 2024-04-24T23:04:45.760 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51120.log.gz 2024-04-24T23:04:45.764 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84842.log 2024-04-24T23:04:45.765 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.84842.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.84842.log.gz 2024-04-24T23:04:45.765 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78936.log 2024-04-24T23:04:45.765 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.78936.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78936.log.gz 2024-04-24T23:04:45.765 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67120.log 2024-04-24T23:04:45.767 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67120.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67120.log.gz 2024-04-24T23:04:45.767 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70866.log 2024-04-24T23:04:45.767 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70866.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70866.log.gz 2024-04-24T23:04:45.767 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47222.log 2024-04-24T23:04:45.768 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.47222.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.47222.log.gz 2024-04-24T23:04:45.768 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70676.log 2024-04-24T23:04:45.769 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.70676.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70676.log.gz 2024-04-24T23:04:45.769 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77175.log 2024-04-24T23:04:45.770 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77175.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.77175.log.gz 2024-04-24T23:04:45.771 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72580.log 2024-04-24T23:04:45.773 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72580.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.72580.log.gz 2024-04-24T23:04:45.773 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61183.log 2024-04-24T23:04:45.774 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61183.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61183.log.gz 2024-04-24T23:04:45.774 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77299.log 2024-04-24T23:04:45.775 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77299.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.77299.log.gz 2024-04-24T23:04:45.775 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65778.log 2024-04-24T23:04:45.776 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.65778.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.65778.log.gz 2024-04-24T23:04:45.776 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56483.log 2024-04-24T23:04:45.777 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56483.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59845.log.gz 2024-04-24T23:04:45.777 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52994.log 2024-04-24T23:04:45.777 INFO:teuthology.orchestra.run.smithi134.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56483.log.gz 2024-04-24T23:04:45.778 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56194.log 2024-04-24T23:04:45.778 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.52994.log: /var/log/ceph/cluster1-client.mirror.56194.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52994.log.gz 2024-04-24T23:04:45.779 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68240.log 2024-04-24T23:04:45.779 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.68240.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.68240.log.gz 2024-04-24T23:04:45.779 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69406.log 2024-04-24T23:04:45.780 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69406.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69406.log.gz 2024-04-24T23:04:45.780 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68087.log 2024-04-24T23:04:45.782 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68087.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68087.log.gz 2024-04-24T23:04:45.782 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62116.log 2024-04-24T23:04:45.783 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62116.log: 73.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62116.log.gz 2024-04-24T23:04:45.784 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48839.log 2024-04-24T23:04:45.784 INFO:teuthology.orchestra.run.smithi134.stderr: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56194.log.gz 2024-04-24T23:04:45.784 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67994.log 2024-04-24T23:04:45.784 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.48839.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.48839.log.gz 2024-04-24T23:04:45.784 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.67994.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80443.log 2024-04-24T23:04:45.785 INFO:teuthology.orchestra.run.smithi134.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.67994.log.gz 2024-04-24T23:04:45.785 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80443.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69978.log 2024-04-24T23:04:45.785 INFO:teuthology.orchestra.run.smithi134.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.80443.log.gz 2024-04-24T23:04:45.785 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52847.log 2024-04-24T23:04:45.786 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.69978.log: /var/log/ceph/cluster2-client.mirror.52847.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69978.log.gz 2024-04-24T23:04:45.787 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59953.log 2024-04-24T23:04:45.787 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59953.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59953.log.gz 2024-04-24T23:04:45.787 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53615.log 2024-04-24T23:04:45.787 INFO:teuthology.orchestra.run.smithi134.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52847.log.gz 2024-04-24T23:04:45.787 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.67643.log 2024-04-24T23:04:45.788 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.53615.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53615.log.gz 2024-04-24T23:04:45.790 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.0.67643.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.67643.log.gz 2024-04-24T23:04:45.790 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72027.log 2024-04-24T23:04:45.790 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64268.log 2024-04-24T23:04:45.791 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.72027.log: /var/log/ceph/cluster1-client.mirror.64268.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.72027.log.gz 2024-04-24T23:04:45.791 INFO:teuthology.orchestra.run.smithi134.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64268.log.gz 2024-04-24T23:04:45.791 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78440.log 2024-04-24T23:04:45.792 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73124.log 2024-04-24T23:04:45.794 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78440.log: /var/log/ceph/cluster1-client.mirror.73124.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.73124.log.gz 2024-04-24T23:04:45.794 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82338.log 2024-04-24T23:04:45.794 INFO:teuthology.orchestra.run.smithi134.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.78440.log.gz 2024-04-24T23:04:45.795 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82338.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.82338.log.gz 2024-04-24T23:04:45.795 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77534.log 2024-04-24T23:04:45.795 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80623.log 2024-04-24T23:04:45.796 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77534.log: /var/log/ceph/cluster1-client.mirror.80623.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80623.log.gz 2024-04-24T23:04:45.796 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56237.log 2024-04-24T23:04:45.798 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.56237.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77534.log.gz 2024-04-24T23:04:45.800 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64315.log 2024-04-24T23:04:45.801 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.64315.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.64315.log.gz 2024-04-24T23:04:45.801 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81201.log 2024-04-24T23:04:45.802 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81201.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.81201.log.gz 2024-04-24T23:04:45.802 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74982.log 2024-04-24T23:04:45.802 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56237.log.gz 2024-04-24T23:04:45.804 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.74982.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.74982.log.gz 2024-04-24T23:04:45.804 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67970.log 2024-04-24T23:04:45.805 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83094.log 2024-04-24T23:04:45.805 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67970.log: /var/log/ceph/cluster2-client.mirror.83094.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67970.log.gz 2024-04-24T23:04:45.806 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58206.log.gz 2024-04-24T23:04:45.806 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83014.log 2024-04-24T23:04:45.807 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78487.log 2024-04-24T23:04:45.807 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83014.log: /var/log/ceph/cluster1-client.mirror.78487.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.78487.log.gz 2024-04-24T23:04:45.807 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68196.log 2024-04-24T23:04:45.809 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68196.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68196.log.gz 2024-04-24T23:04:45.809 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60858.log 2024-04-24T23:04:45.809 INFO:teuthology.orchestra.run.smithi134.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.83014.log.gz 2024-04-24T23:04:45.810 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60858.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60858.log.gz 2024-04-24T23:04:45.810 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53663.log 2024-04-24T23:04:45.810 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54518.log 2024-04-24T23:04:45.811 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.53663.log: /var/log/ceph/cluster1-client.mirror.54518.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53663.log.gz 2024-04-24T23:04:45.811 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66563.log 2024-04-24T23:04:45.811 INFO:teuthology.orchestra.run.smithi134.stderr: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54518.log.gz 2024-04-24T23:04:45.811 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57237.log 2024-04-24T23:04:45.812 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66563.log: /var/log/ceph/cluster2-client.mirror.57237.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66563.log.gz 2024-04-24T23:04:45.818 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51258.log 2024-04-24T23:04:45.818 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.51258.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46928.log 2024-04-24T23:04:45.819 INFO:teuthology.orchestra.run.smithi134.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.51258.log.gz 2024-04-24T23:04:45.819 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.46928.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.46928.log.gz 2024-04-24T23:04:45.819 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76070.log 2024-04-24T23:04:45.820 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76070.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76070.log.gz 2024-04-24T23:04:45.820 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76554.log 2024-04-24T23:04:45.822 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76554.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75049.log 89.2% 2024-04-24T23:04:45.822 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.76554.log.gz 2024-04-24T23:04:45.824 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75049.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75049.log.gz 2024-04-24T23:04:45.825 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.84617.log 2024-04-24T23:04:45.826 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.1.84617.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.1.84617.log.gz 2024-04-24T23:04:45.826 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55632.log 2024-04-24T23:04:45.830 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55632.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70065.log 2024-04-24T23:04:45.833 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70065.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60172.log.gz 2024-04-24T23:04:45.833 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55632.log.gz 2024-04-24T23:04:45.833 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50460.log 2024-04-24T23:04:45.833 INFO:teuthology.orchestra.run.smithi134.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70065.log.gz 2024-04-24T23:04:45.833 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72138.log 2024-04-24T23:04:45.834 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.50460.log: /var/log/ceph/cluster2-client.mirror.72138.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50460.log.gz 2024-04-24T23:04:45.834 INFO:teuthology.orchestra.run.smithi134.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72138.log.gz 2024-04-24T23:04:45.834 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61417.log 2024-04-24T23:04:45.834 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59379.log 2024-04-24T23:04:45.835 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61417.log: /var/log/ceph/cluster1-client.mirror.59379.log: 86.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59379.log.gz 2024-04-24T23:04:45.835 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79566.log 2024-04-24T23:04:45.835 INFO:teuthology.orchestra.run.smithi134.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61417.log.gz 2024-04-24T23:04:45.836 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55847.log 2024-04-24T23:04:45.837 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79566.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79566.log.gz 2024-04-24T23:04:45.837 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55847.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75711.log 2024-04-24T23:04:45.837 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75711.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75711.log.gz 2024-04-24T23:04:45.837 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47514.log 2024-04-24T23:04:45.838 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65557.log 2024-04-24T23:04:45.838 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.47514.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.47514.log.gz 2024-04-24T23:04:45.838 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48116.log 2024-04-24T23:04:45.839 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.65557.log: /var/log/ceph/cluster1-client.mirror.48116.log: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.48116.log.gz 2024-04-24T23:04:45.839 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47318.log 2024-04-24T23:04:45.840 INFO:teuthology.orchestra.run.smithi134.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65557.log.gz 2024-04-24T23:04:45.840 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.47318.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.47318.log.gz 2024-04-24T23:04:45.840 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65968.log 2024-04-24T23:04:45.841 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.65968.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65968.log.gz 2024-04-24T23:04:45.841 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61310.log 2024-04-24T23:04:45.842 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55847.log.gz 2024-04-24T23:04:45.842 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61310.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61310.log.gz 2024-04-24T23:04:45.842 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62921.log 2024-04-24T23:04:45.843 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57012.log 2024-04-24T23:04:45.844 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.62921.log: /var/log/ceph/cluster2-client.mirror.57012.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57012.log.gz 2024-04-24T23:04:45.844 INFO:teuthology.orchestra.run.smithi134.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.62921.log.gz 2024-04-24T23:04:45.844 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54226.log 2024-04-24T23:04:45.844 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49481.log 2024-04-24T23:04:45.845 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54226.log: 86.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54226.log.gz 2024-04-24T23:04:45.849 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49481.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62065.log 2024-04-24T23:04:45.849 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56303.log 2024-04-24T23:04:45.849 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.62065.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62065.log.gz 2024-04-24T23:04:45.850 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70986.log 2024-04-24T23:04:45.850 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56303.log: /var/log/ceph/cluster1-client.mirror.70986.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56303.log.gz 2024-04-24T23:04:45.851 INFO:teuthology.orchestra.run.smithi134.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70986.log.gz 2024-04-24T23:04:45.851 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61630.log 2024-04-24T23:04:45.851 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79498.log 2024-04-24T23:04:45.852 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61630.log: /var/log/ceph/cluster1-client.mirror.79498.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.79498.log.gz 2024-04-24T23:04:45.852 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74146.log 2024-04-24T23:04:45.854 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74146.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74146.log.gz 2024-04-24T23:04:45.854 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67475.log 2024-04-24T23:04:45.856 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67475.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67475.log.gz 2024-04-24T23:04:45.857 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.82403.log 2024-04-24T23:04:45.862 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.0.82403.log: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.0.82403.log.gz 2024-04-24T23:04:45.862 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75111.log 2024-04-24T23:04:45.865 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75111.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75111.log.gz 2024-04-24T23:04:45.865 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80668.log 2024-04-24T23:04:45.866 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80668.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.80668.log.gz 2024-04-24T23:04:45.866 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61687.log 2024-04-24T23:04:45.868 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61687.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61687.log.gz 2024-04-24T23:04:45.868 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55495.log 2024-04-24T23:04:45.874 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55495.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55495.log.gz 2024-04-24T23:04:45.874 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77812.log 2024-04-24T23:04:45.875 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77812.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77812.log.gz 2024-04-24T23:04:45.875 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61667.log 2024-04-24T23:04:45.877 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61667.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61667.log.gz 2024-04-24T23:04:45.877 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72842.log 2024-04-24T23:04:45.879 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72842.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.72842.log.gz 2024-04-24T23:04:45.880 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61347.log 2024-04-24T23:04:45.881 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61347.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61347.log.gz 2024-04-24T23:04:45.881 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.71437.log 2024-04-24T23:04:45.885 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.0.71437.log: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.71437.log.gz 2024-04-24T23:04:45.885 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59639.log 2024-04-24T23:04:45.891 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59639.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48700.log 2024-04-24T23:04:45.907 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.48700.log: 58.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48700.log.gz 2024-04-24T23:04:45.918 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65516.log 2024-04-24T23:04:45.929 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.65516.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.65516.log.gz 2024-04-24T23:04:45.929 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84617.log 2024-04-24T23:04:45.938 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.84617.log: 81.5% -- replaced with /var/log/ceph/cluster1-client.mirror.84617.log.gz 2024-04-24T23:04:45.949 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48751.log 2024-04-24T23:04:45.960 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.48751.log: 54.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48751.log.gz 2024-04-24T23:04:45.960 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75401.log 2024-04-24T23:04:45.971 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75401.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75401.log.gz 2024-04-24T23:04:45.971 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78666.log 2024-04-24T23:04:45.982 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.78666.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78666.log.gz 2024-04-24T23:04:45.982 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55515.log 2024-04-24T23:04:45.993 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.55515.log: 94.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49481.log.gz 2024-04-24T23:04:45.993 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60680.log 2024-04-24T23:04:45.993 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55515.log.gz 2024-04-24T23:04:45.994 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60680.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60680.log.gz 2024-04-24T23:04:45.994 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58808.log 2024-04-24T23:04:45.995 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58808.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58808.log.gz 2024-04-24T23:04:45.996 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64753.log 2024-04-24T23:04:45.997 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64753.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64753.log.gz 2024-04-24T23:04:45.997 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47076.log 2024-04-24T23:04:45.998 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.47076.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.47076.log.gz 2024-04-24T23:04:45.998 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85128.log 2024-04-24T23:04:45.998 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.85128.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85128.log.gz 2024-04-24T23:04:45.998 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58245.log 2024-04-24T23:04:46.000 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58245.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58245.log.gz 2024-04-24T23:04:46.000 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54893.log 2024-04-24T23:04:46.002 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54893.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54893.log.gz 2024-04-24T23:04:46.002 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62764.log 2024-04-24T23:04:46.003 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62764.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62764.log.gz 2024-04-24T23:04:46.003 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51354.log 2024-04-24T23:04:46.014 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.51354.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71366.log 2024-04-24T23:04:46.017 INFO:teuthology.orchestra.run.smithi134.stderr: 95.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51354.log.gz 2024-04-24T23:04:46.018 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71366.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71366.log.gz 2024-04-24T23:04:46.019 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62705.log 2024-04-24T23:04:46.020 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62705.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62705.log.gz 2024-04-24T23:04:46.020 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66000.log 2024-04-24T23:04:46.021 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.66000.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.66000.log.gz 2024-04-24T23:04:46.021 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69192.log 2024-04-24T23:04:46.022 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.69192.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69192.log.gz 2024-04-24T23:04:46.022 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63744.log 2024-04-24T23:04:46.023 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63744.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63744.log.gz 2024-04-24T23:04:46.023 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76823.log 2024-04-24T23:04:46.024 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76823.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76823.log.gz 2024-04-24T23:04:46.024 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56105.log 2024-04-24T23:04:46.025 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.56105.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56105.log.gz 2024-04-24T23:04:46.025 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49112.log 2024-04-24T23:04:46.026 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.49112.log: 67.2% -- replaced with /var/log/ceph/cluster2-client.admin.49112.log.gz 2024-04-24T23:04:46.026 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61205.log 2024-04-24T23:04:46.030 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61205.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.81451.log 2024-04-24T23:04:46.059 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.1.81451.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76740.log 2024-04-24T23:04:46.060 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57237.log.gz 2024-04-24T23:04:46.060 INFO:teuthology.orchestra.run.smithi134.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.1.81451.log.gz 2024-04-24T23:04:46.064 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76740.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71716.log 2024-04-24T23:04:46.064 INFO:teuthology.orchestra.run.smithi134.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76740.log.gz 2024-04-24T23:04:46.066 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71716.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71716.log.gz 2024-04-24T23:04:46.066 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63332.log 2024-04-24T23:04:46.067 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.63332.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63332.log.gz 2024-04-24T23:04:46.067 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73622.log 2024-04-24T23:04:46.068 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73622.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73622.log.gz 2024-04-24T23:04:46.069 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60801.log 2024-04-24T23:04:46.080 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60801.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57354.log 2024-04-24T23:04:46.102 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57354.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57354.log.gz 2024-04-24T23:04:46.102 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55144.log 2024-04-24T23:04:46.109 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55144.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55144.log.gz 2024-04-24T23:04:46.119 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58729.log 2024-04-24T23:04:46.131 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58729.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62546.log 2024-04-24T23:04:46.131 INFO:teuthology.orchestra.run.smithi134.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58729.log.gz 2024-04-24T23:04:46.142 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62546.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62546.log.gz 2024-04-24T23:04:46.142 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59865.log 2024-04-24T23:04:46.155 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59865.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59865.log.gz 2024-04-24T23:04:46.159 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61630.log.gz 2024-04-24T23:04:46.160 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64969.log 2024-04-24T23:04:46.161 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.64969.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59010.log.gz 2024-04-24T23:04:46.161 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61146.log 2024-04-24T23:04:46.162 INFO:teuthology.orchestra.run.smithi134.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.64969.log.gz 2024-04-24T23:04:46.162 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80812.log 2024-04-24T23:04:46.162 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61146.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61146.log.gz 2024-04-24T23:04:46.164 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80812.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.80812.log.gz 2024-04-24T23:04:46.164 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76761.log 2024-04-24T23:04:46.164 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81016.log 2024-04-24T23:04:46.165 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76761.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.76761.log.gz 2024-04-24T23:04:46.165 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.81016.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.81016.log.gz 2024-04-24T23:04:46.166 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75587.log 2024-04-24T23:04:46.167 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77369.log 2024-04-24T23:04:46.167 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61126.log.gz 2024-04-24T23:04:46.167 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75587.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75587.log.gz 2024-04-24T23:04:46.167 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77369.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77369.log.gz 2024-04-24T23:04:46.168 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67756.log 2024-04-24T23:04:46.168 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79543.log 2024-04-24T23:04:46.169 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.67756.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.67756.log.gz 2024-04-24T23:04:46.169 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79543.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79543.log.gz 2024-04-24T23:04:46.169 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72376.log 2024-04-24T23:04:46.169 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82813.log 2024-04-24T23:04:46.170 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.72376.log: /var/log/ceph/cluster2-client.mirror.82813.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.82813.log.gz 2024-04-24T23:04:46.170 INFO:teuthology.orchestra.run.smithi134.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72376.log.gz 2024-04-24T23:04:46.170 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70745.log 2024-04-24T23:04:46.172 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70745.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70745.log.gz 2024-04-24T23:04:46.172 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71935.log 2024-04-24T23:04:46.173 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54326.log 2024-04-24T23:04:46.174 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71935.log: /var/log/ceph/cluster1-client.admin.54326.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.54326.log.gz 2024-04-24T23:04:46.174 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62526.log 2024-04-24T23:04:46.174 INFO:teuthology.orchestra.run.smithi134.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71935.log.gz 2024-04-24T23:04:46.175 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62526.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62526.log.gz 2024-04-24T23:04:46.175 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84433.log 2024-04-24T23:04:46.176 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43288.log 2024-04-24T23:04:46.176 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.84433.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.84433.log.gz 2024-04-24T23:04:46.177 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.43288.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.43288.log.gz 2024-04-24T23:04:46.177 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79903.log 2024-04-24T23:04:46.177 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76389.log 2024-04-24T23:04:46.177 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79903.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79903.log.gz 2024-04-24T23:04:46.178 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76389.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.76389.log.gz 2024-04-24T23:04:46.178 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69785.log 2024-04-24T23:04:46.178 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76699.log 2024-04-24T23:04:46.180 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69785.log: /var/log/ceph/cluster2-client.mirror.76699.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76699.log.gz 2024-04-24T23:04:46.180 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51798.log 2024-04-24T23:04:46.180 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.51798.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69785.log.gz 2024-04-24T23:04:46.180 INFO:teuthology.orchestra.run.smithi134.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.51798.log.gz 2024-04-24T23:04:46.180 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51490.log 2024-04-24T23:04:46.181 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74205.log 2024-04-24T23:04:46.181 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51490.log: /var/log/ceph/cluster2-client.mirror.74205.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51490.log.gz 2024-04-24T23:04:46.182 INFO:teuthology.orchestra.run.smithi134.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74205.log.gz 2024-04-24T23:04:46.182 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68591.log 2024-04-24T23:04:46.182 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54119.log 2024-04-24T23:04:46.184 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.68591.log: /var/log/ceph/cluster2-client.mirror.54119.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.68591.log.gz 2024-04-24T23:04:46.188 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54119.log.gz 2024-04-24T23:04:46.188 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53791.log 2024-04-24T23:04:46.189 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80993.log 2024-04-24T23:04:46.199 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53791.log: /var/log/ceph/cluster1-client.mirror.80993.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.80993.log.gz 2024-04-24T23:04:46.199 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48069.log 2024-04-24T23:04:46.199 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.48069.log: 74.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48069.log.gz 2024-04-24T23:04:46.199 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85899.log 2024-04-24T23:04:46.199 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.85899.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85899.log.gz 2024-04-24T23:04:46.199 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67100.log 2024-04-24T23:04:46.199 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67100.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67100.log.gz 2024-04-24T23:04:46.199 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73225.log 2024-04-24T23:04:46.199 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73225.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73225.log.gz 2024-04-24T23:04:46.199 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59823.log 2024-04-24T23:04:46.199 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59823.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59823.log.gz 2024-04-24T23:04:46.200 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.83320.log 2024-04-24T23:04:46.202 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.3.83320.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.3.83320.log.gz 2024-04-24T23:04:46.202 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56583.log 2024-04-24T23:04:46.203 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56583.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56583.log.gz 2024-04-24T23:04:46.203 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50294.log 2024-04-24T23:04:46.204 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.50294.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.50294.log.gz 2024-04-24T23:04:46.204 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70628.log 2024-04-24T23:04:46.205 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70628.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70628.log.gz 2024-04-24T23:04:46.205 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-04-24T23:04:46.213 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50584.log 2024-04-24T23:04:46.214 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.50584.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50584.log.gz 2024-04-24T23:04:46.214 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73559.log 2024-04-24T23:04:46.215 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.73559.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.73559.log.gz 2024-04-24T23:04:46.215 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61225.log 2024-04-24T23:04:46.216 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61225.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61225.log.gz 2024-04-24T23:04:46.216 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61104.log 2024-04-24T23:04:46.218 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61104.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61104.log.gz 2024-04-24T23:04:46.218 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75029.log 2024-04-24T23:04:46.219 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75029.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75029.log.gz 2024-04-24T23:04:46.219 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60700.log 2024-04-24T23:04:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60700.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60700.log.gz 2024-04-24T23:04:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53287.log 2024-04-24T23:04:46.221 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53287.log: 83.1% -- replaced with /var/log/ceph/cluster2-client.mirror.53287.log.gz 2024-04-24T23:04:46.221 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75566.log 2024-04-24T23:04:46.223 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75566.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75566.log.gz 2024-04-24T23:04:46.223 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.54749.log 2024-04-24T23:04:46.225 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.2.54749.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.83094.log.gz 2024-04-24T23:04:46.225 INFO:teuthology.orchestra.run.smithi134.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.2.54749.log.gz 2024-04-24T23:04:46.225 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-04-24T23:04:46.226 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69327.log 2024-04-24T23:04:46.227 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-osd.2.log: /var/log/ceph/cluster2-client.mirror.69327.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69327.log.gz 2024-04-24T23:04:46.227 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56691.log 2024-04-24T23:04:46.229 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56691.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56691.log.gz 2024-04-24T23:04:46.229 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53915.log 2024-04-24T23:04:46.230 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53915.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53915.log.gz 2024-04-24T23:04:46.230 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78598.log 2024-04-24T23:04:46.232 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.78598.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.78598.log.gz 2024-04-24T23:04:46.232 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83688.log 2024-04-24T23:04:46.233 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.83688.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.83688.log.gz 2024-04-24T23:04:46.233 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60821.log 2024-04-24T23:04:46.234 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60821.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60821.log.gz 2024-04-24T23:04:46.234 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83831.log 2024-04-24T23:04:46.235 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83831.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.83831.log.gz 2024-04-24T23:04:46.235 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49253.log 2024-04-24T23:04:46.236 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49253.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49253.log.gz 2024-04-24T23:04:46.236 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65841.log 2024-04-24T23:04:46.238 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65841.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65841.log.gz 2024-04-24T23:04:46.238 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53639.log 2024-04-24T23:04:46.239 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.53639.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53639.log.gz 2024-04-24T23:04:46.239 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65341.log 2024-04-24T23:04:46.241 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65341.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65341.log.gz 2024-04-24T23:04:46.241 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75752.log 2024-04-24T23:04:46.242 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75752.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75752.log.gz 2024-04-24T23:04:46.242 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75793.log 2024-04-24T23:04:46.245 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75793.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75793.log.gz 2024-04-24T23:04:46.245 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55343.log 2024-04-24T23:04:46.261 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.55343.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54694.log.gz 2024-04-24T23:04:46.261 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51051.log 2024-04-24T23:04:46.261 INFO:teuthology.orchestra.run.smithi134.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55343.log.gz 2024-04-24T23:04:46.266 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51051.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79926.log 2024-04-24T23:04:46.267 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51051.log.gz 2024-04-24T23:04:46.268 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79926.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79926.log.gz 2024-04-24T23:04:46.268 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74384.log 2024-04-24T23:04:46.269 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74384.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74384.log.gz 2024-04-24T23:04:46.269 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76194.log 2024-04-24T23:04:46.270 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76194.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76194.log.gz 2024-04-24T23:04:46.270 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53244.log 2024-04-24T23:04:46.272 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53244.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53244.log.gz 2024-04-24T23:04:46.272 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76595.log 2024-04-24T23:04:46.275 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76595.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76595.log.gz 2024-04-24T23:04:46.275 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48734.log 2024-04-24T23:04:46.276 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.48734.log: 54.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48734.log.gz 2024-04-24T23:04:46.276 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81822.log 2024-04-24T23:04:46.277 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81822.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.81822.log.gz 2024-04-24T23:04:46.277 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62832.log 2024-04-24T23:04:46.278 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.62832.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62832.log.gz 2024-04-24T23:04:46.278 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67576.log 2024-04-24T23:04:46.279 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67576.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67576.log.gz 2024-04-24T23:04:46.279 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58643.log 2024-04-24T23:04:46.280 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58643.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58643.log.gz 2024-04-24T23:04:46.280 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60722.log 2024-04-24T23:04:46.285 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60722.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60996.log 2024-04-24T23:04:46.287 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60996.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60996.log.gz 2024-04-24T23:04:46.292 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43335.log 2024-04-24T23:04:46.293 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.43335.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.43335.log.gz 2024-04-24T23:04:46.298 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60069.log 2024-04-24T23:04:46.310 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60069.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60069.log.gz 2024-04-24T23:04:46.310 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80511.log 2024-04-24T23:04:46.311 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80511.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.80511.log.gz 2024-04-24T23:04:46.311 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63635.log 2024-04-24T23:04:46.314 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.63635.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63635.log.gz 2024-04-24T23:04:46.314 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52970.log 2024-04-24T23:04:46.328 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.52970.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52970.log.gz 2024-04-24T23:04:46.328 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80533.log 2024-04-24T23:04:46.329 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80533.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80533.log.gz 2024-04-24T23:04:46.329 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71108.log 2024-04-24T23:04:46.331 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71108.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71108.log.gz 2024-04-24T23:04:46.331 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57135.log 2024-04-24T23:04:46.332 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57135.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57135.log.gz 2024-04-24T23:04:46.332 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60959.log 2024-04-24T23:04:46.343 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60959.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52310.log 2024-04-24T23:04:46.359 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.52310.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.52310.log.gz 2024-04-24T23:04:46.360 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59902.log 2024-04-24T23:04:46.370 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59902.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59902.log.gz 2024-04-24T23:04:46.370 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80421.log 2024-04-24T23:04:46.386 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80421.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.80421.log.gz 2024-04-24T23:04:46.386 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51824.log 2024-04-24T23:04:46.408 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.51824.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51824.log.gz 2024-04-24T23:04:46.408 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70507.log 2024-04-24T23:04:46.422 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70507.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82714.log 2024-04-24T23:04:46.422 INFO:teuthology.orchestra.run.smithi134.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70507.log.gz 2024-04-24T23:04:46.433 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.82714.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82714.log.gz 2024-04-24T23:04:46.433 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80398.log 2024-04-24T23:04:46.437 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61205.log.gz 2024-04-24T23:04:46.438 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80398.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.80398.log.gz 2024-04-24T23:04:46.438 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53815.log 2024-04-24T23:04:46.439 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.53815.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.53815.log.gz 2024-04-24T23:04:46.439 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50987.log 2024-04-24T23:04:46.440 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.50987.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50987.log.gz 2024-04-24T23:04:46.441 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55472.log 2024-04-24T23:04:46.444 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.55472.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52664.log 2024-04-24T23:04:46.455 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55472.log.gz 2024-04-24T23:04:46.455 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52664.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52664.log.gz 2024-04-24T23:04:46.455 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67216.log 2024-04-24T23:04:46.455 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.67216.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.67216.log.gz 2024-04-24T23:04:46.455 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52127.log 2024-04-24T23:04:46.461 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52127.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52127.log.gz 2024-04-24T23:04:46.461 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57609.log 2024-04-24T23:04:46.461 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57609.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57609.log.gz 2024-04-24T23:04:46.461 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49163.log 2024-04-24T23:04:46.461 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49163.log: 3.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49163.log.gz 2024-04-24T23:04:46.461 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58606.log 2024-04-24T23:04:46.462 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58606.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58606.log.gz 2024-04-24T23:04:46.462 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85948.log 2024-04-24T23:04:46.462 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.85948.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85948.log.gz 2024-04-24T23:04:46.462 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71875.log 2024-04-24T23:04:46.462 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71875.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60801.log.gz 2024-04-24T23:04:46.462 INFO:teuthology.orchestra.run.smithi134.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71875.log.gz 2024-04-24T23:04:46.462 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69002.log 2024-04-24T23:04:46.463 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.69002.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66714.log 2024-04-24T23:04:46.463 INFO:teuthology.orchestra.run.smithi134.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.69002.log.gz 2024-04-24T23:04:46.463 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.66714.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.66714.log.gz 2024-04-24T23:04:46.463 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68764.log 2024-04-24T23:04:46.464 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.68764.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68764.log.gz 2024-04-24T23:04:46.464 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55121.log 2024-04-24T23:04:46.470 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.55121.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55121.log.gz 2024-04-24T23:04:46.470 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78621.log 2024-04-24T23:04:46.471 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.78621.log: 85.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78621.log.gz 2024-04-24T23:04:46.471 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58888.log 2024-04-24T23:04:46.472 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58888.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59639.log.gz 2024-04-24T23:04:46.473 INFO:teuthology.orchestra.run.smithi134.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58888.log.gz 2024-04-24T23:04:46.473 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67643.log 2024-04-24T23:04:46.473 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71007.log 2024-04-24T23:04:46.473 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67643.log: 82.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67643.log.gz 2024-04-24T23:04:46.475 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71007.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71007.log.gz 2024-04-24T23:04:46.475 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57726.log 2024-04-24T23:04:46.476 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84350.log 2024-04-24T23:04:46.476 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57726.log: /var/log/ceph/cluster1-client.mirror.84350.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57726.log.gz 2024-04-24T23:04:46.476 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84350.log.gz 2024-04-24T23:04:46.477 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65444.log 2024-04-24T23:04:46.477 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.65444.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65444.log.gz 2024-04-24T23:04:46.478 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-04-24T23:04:46.478 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67891.log 2024-04-24T23:04:46.485 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-osd.1.log: /var/log/ceph/cluster2-client.mirror.67891.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67891.log.gz 2024-04-24T23:04:46.485 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83853.log 2024-04-24T23:04:46.496 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83853.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85360.log 2024-04-24T23:04:46.496 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.85360.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85360.log.gz 2024-04-24T23:04:46.506 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67849.log 2024-04-24T23:04:46.506 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67849.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67849.log.gz 2024-04-24T23:04:46.509 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74249.log 2024-04-24T23:04:46.512 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.74249.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74249.log.gz 2024-04-24T23:04:46.515 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78006.log 2024-04-24T23:04:46.518 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78006.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.78006.log.gz 2024-04-24T23:04:46.518 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82122.log 2024-04-24T23:04:46.519 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.82122.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82122.log.gz 2024-04-24T23:04:46.519 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73943.log 2024-04-24T23:04:46.521 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73943.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73943.log.gz 2024-04-24T23:04:46.521 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78981.log 2024-04-24T23:04:46.522 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.78981.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.78981.log.gz 2024-04-24T23:04:46.522 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57706.log 2024-04-24T23:04:46.524 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57706.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57706.log.gz 2024-04-24T23:04:46.524 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77112.log 2024-04-24T23:04:46.525 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77112.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77112.log.gz 2024-04-24T23:04:46.525 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61369.log 2024-04-24T23:04:46.536 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61369.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60838.log 2024-04-24T23:04:46.552 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60838.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60838.log.gz 2024-04-24T23:04:46.552 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65162.log 2024-04-24T23:04:46.552 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53791.log.gz 2024-04-24T23:04:46.553 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65162.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65162.log.gz 2024-04-24T23:04:46.554 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53170.log 2024-04-24T23:04:46.564 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.53170.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.82580.log 2024-04-24T23:04:46.585 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.1.82580.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.82580.log.gz 2024-04-24T23:04:46.586 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65754.log 2024-04-24T23:04:46.586 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60959.log.gz 2024-04-24T23:04:46.591 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.65754.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65754.log.gz 2024-04-24T23:04:46.591 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74583.log 2024-04-24T23:04:46.592 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74583.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74583.log.gz 2024-04-24T23:04:46.592 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78643.log 2024-04-24T23:04:46.596 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60722.log.gz 2024-04-24T23:04:46.603 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77320.log 2024-04-24T23:04:46.604 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.78643.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.78643.log.gz 2024-04-24T23:04:46.604 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77320.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77320.log.gz 2024-04-24T23:04:46.604 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61284.log 2024-04-24T23:04:46.605 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61284.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66801.log 2024-04-24T23:04:46.606 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66801.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66801.log.gz 2024-04-24T23:04:46.606 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69764.log 2024-04-24T23:04:46.608 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.69764.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69764.log.gz 2024-04-24T23:04:46.608 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64452.log 2024-04-24T23:04:46.609 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.64452.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64452.log.gz 2024-04-24T23:04:46.609 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80646.log 2024-04-24T23:04:46.610 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80646.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.80646.log.gz 2024-04-24T23:04:46.610 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46730.log 2024-04-24T23:04:46.611 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.46730.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.46730.log.gz 2024-04-24T23:04:46.611 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80218.log 2024-04-24T23:04:46.612 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80218.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.80218.log.gz 2024-04-24T23:04:46.612 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50774.log 2024-04-24T23:04:46.614 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.50774.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.50774.log.gz 2024-04-24T23:04:46.614 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52193.log 2024-04-24T23:04:46.615 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52193.log: 81.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52193.log.gz 2024-04-24T23:04:46.615 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53191.log 2024-04-24T23:04:46.616 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.53191.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.53191.log.gz 2024-04-24T23:04:46.616 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53502.log 2024-04-24T23:04:46.617 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.53502.log: 79.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53502.log.gz 2024-04-24T23:04:46.617 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56732.log 2024-04-24T23:04:46.620 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56732.log: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.56732.log.gz 2024-04-24T23:04:46.620 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68502.log 2024-04-24T23:04:46.622 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.68502.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68502.log.gz 2024-04-24T23:04:46.622 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74869.log 2024-04-24T23:04:46.623 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.74869.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74869.log.gz 2024-04-24T23:04:46.722 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75091.log 2024-04-24T23:04:46.728 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75091.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.75091.log.gz 2024-04-24T23:04:46.728 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59067.log 2024-04-24T23:04:46.728 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59067.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59067.log.gz 2024-04-24T23:04:46.728 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79881.log 2024-04-24T23:04:46.729 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79881.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.79881.log.gz 2024-04-24T23:04:46.729 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64244.log 2024-04-24T23:04:46.729 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64244.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64244.log.gz 2024-04-24T23:04:46.729 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66622.log 2024-04-24T23:04:46.729 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66622.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66622.log.gz 2024-04-24T23:04:46.729 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63181.log 2024-04-24T23:04:46.729 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63181.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63181.log.gz 2024-04-24T23:04:46.729 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57869.log 2024-04-24T23:04:46.729 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57869.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57869.log.gz 2024-04-24T23:04:46.729 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83161.log 2024-04-24T23:04:46.729 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83161.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83161.log.gz 2024-04-24T23:04:46.729 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48548.log 2024-04-24T23:04:46.729 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.48548.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48548.log.gz 2024-04-24T23:04:46.729 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63962.log 2024-04-24T23:04:46.729 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63962.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63962.log.gz 2024-04-24T23:04:46.730 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50915.log 2024-04-24T23:04:46.730 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.50915.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50915.log.gz 2024-04-24T23:04:46.730 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58147.log 2024-04-24T23:04:46.730 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58147.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58147.log.gz 2024-04-24T23:04:46.730 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79003.log 2024-04-24T23:04:46.730 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79003.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79003.log.gz 2024-04-24T23:04:46.730 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79746.log 2024-04-24T23:04:46.730 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79746.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.79746.log.gz 2024-04-24T23:04:46.730 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66045.log 2024-04-24T23:04:46.730 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66045.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66045.log.gz 2024-04-24T23:04:46.730 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58265.log 2024-04-24T23:04:46.730 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58265.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58265.log.gz 2024-04-24T23:04:46.730 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85703.log 2024-04-24T23:04:46.730 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.85703.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85703.log.gz 2024-04-24T23:04:46.730 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51997.log 2024-04-24T23:04:46.731 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.51997.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.51997.log.gz 2024-04-24T23:04:46.731 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70787.log 2024-04-24T23:04:46.731 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70787.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70787.log.gz 2024-04-24T23:04:46.731 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63377.log 2024-04-24T23:04:46.731 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63377.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63377.log.gz 2024-04-24T23:04:46.731 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63397.log 2024-04-24T23:04:46.731 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.63397.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63397.log.gz 2024-04-24T23:04:46.731 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50528.log 2024-04-24T23:04:46.731 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.50528.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.50528.log.gz 2024-04-24T23:04:46.731 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70289.log 2024-04-24T23:04:46.731 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.70289.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70289.log.gz 2024-04-24T23:04:46.731 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60304.log 2024-04-24T23:04:46.731 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60304.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60304.log.gz 2024-04-24T23:04:46.731 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76575.log 2024-04-24T23:04:46.731 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76575.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.76575.log.gz 2024-04-24T23:04:46.732 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83320.log 2024-04-24T23:04:46.732 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.83320.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.83320.log.gz 2024-04-24T23:04:46.732 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.76278.log 2024-04-24T23:04:46.732 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.2.76278.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73797.log 2024-04-24T23:04:46.732 INFO:teuthology.orchestra.run.smithi134.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.2.76278.log.gz 2024-04-24T23:04:46.732 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.73797.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.73797.log.gz 2024-04-24T23:04:46.732 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47074.log 2024-04-24T23:04:46.732 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.47074.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.47074.log.gz 2024-04-24T23:04:46.732 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56260.log 2024-04-24T23:04:46.732 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56260.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56260.log.gz 2024-04-24T23:04:46.732 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56663.log 2024-04-24T23:04:46.732 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56663.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56663.log.gz 2024-04-24T23:04:46.732 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76616.log 2024-04-24T23:04:46.732 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76616.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76616.log.gz 2024-04-24T23:04:46.732 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83140.log 2024-04-24T23:04:46.733 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.83140.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.83140.log.gz 2024-04-24T23:04:46.733 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49197.log 2024-04-24T23:04:46.733 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.49197.log: 57.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49197.log.gz 2024-04-24T23:04:46.733 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52707.log 2024-04-24T23:04:46.733 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52707.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52707.log.gz 2024-04-24T23:04:46.733 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80308.log 2024-04-24T23:04:46.733 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80308.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.80308.log.gz 2024-04-24T23:04:46.733 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69089.log 2024-04-24T23:04:46.733 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69089.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69089.log.gz 2024-04-24T23:04:46.733 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62570.log 2024-04-24T23:04:46.733 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.62570.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62570.log.gz 2024-04-24T23:04:46.733 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80061.log 2024-04-24T23:04:46.733 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80061.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80061.log.gz 2024-04-24T23:04:46.733 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70938.log 2024-04-24T23:04:46.733 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.70938.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70938.log.gz 2024-04-24T23:04:46.734 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62784.log 2024-04-24T23:04:46.734 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62784.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62784.log.gz 2024-04-24T23:04:46.734 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48253.log 2024-04-24T23:04:46.734 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.48253.log: 74.1% -- replaced with /var/log/ceph/cluster1-client.mirror.48253.log.gz 2024-04-24T23:04:46.734 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61327.log 2024-04-24T23:04:46.734 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61327.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61327.log.gz 2024-04-24T23:04:46.734 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75235.log 2024-04-24T23:04:46.734 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75235.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75235.log.gz 2024-04-24T23:04:46.734 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71049.log 2024-04-24T23:04:46.734 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.71049.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71049.log.gz 2024-04-24T23:04:46.734 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68672.log 2024-04-24T23:04:46.734 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68672.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68672.log.gz 2024-04-24T23:04:46.734 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62229.log 2024-04-24T23:04:46.734 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62229.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62229.log.gz 2024-04-24T23:04:46.734 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80466.log 2024-04-24T23:04:46.735 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80466.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.80466.log.gz 2024-04-24T23:04:46.735 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85032.log 2024-04-24T23:04:46.735 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.85032.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85032.log.gz 2024-04-24T23:04:46.735 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66521.log 2024-04-24T23:04:46.735 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.66521.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66521.log.gz 2024-04-24T23:04:46.735 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59277.log 2024-04-24T23:04:46.735 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59277.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59277.log.gz 2024-04-24T23:04:46.735 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61163.log 2024-04-24T23:04:46.735 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61163.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61163.log.gz 2024-04-24T23:04:46.735 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76235.log 2024-04-24T23:04:46.735 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76235.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76235.log.gz 2024-04-24T23:04:46.735 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65900.log 2024-04-24T23:04:46.735 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65900.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65900.log.gz 2024-04-24T23:04:46.735 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63570.log 2024-04-24T23:04:46.735 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.63570.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63570.log.gz 2024-04-24T23:04:46.736 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63139.log 2024-04-24T23:04:46.736 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63139.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63139.log.gz 2024-04-24T23:04:46.736 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70370.log 2024-04-24T23:04:46.736 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70370.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70370.log.gz 2024-04-24T23:04:46.736 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57928.log 2024-04-24T23:04:46.736 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57928.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60012.log 2024-04-24T23:04:46.756 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60012.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64165.log 2024-04-24T23:04:46.768 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64165.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64165.log.gz 2024-04-24T23:04:46.768 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60032.log 2024-04-24T23:04:46.779 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60032.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60032.log.gz 2024-04-24T23:04:46.779 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74941.log 2024-04-24T23:04:46.793 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.74941.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.74941.log.gz 2024-04-24T23:04:46.793 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54630.log 2024-04-24T23:04:46.807 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54630.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58228.log 2024-04-24T23:04:46.812 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54630.log.gz 2024-04-24T23:04:46.820 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61369.log.gz 2024-04-24T23:04:46.821 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58228.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58228.log.gz 2024-04-24T23:04:46.821 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55976.log 2024-04-24T23:04:46.827 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55976.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55976.log.gz 2024-04-24T23:04:46.827 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63982.log 2024-04-24T23:04:46.828 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63982.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63982.log.gz 2024-04-24T23:04:46.828 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60471.log 2024-04-24T23:04:46.828 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60471.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.60471.log.gz 2024-04-24T23:04:46.828 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84254.log 2024-04-24T23:04:46.829 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.84254.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.84254.log.gz 2024-04-24T23:04:46.829 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60321.log 2024-04-24T23:04:46.831 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60321.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60321.log.gz 2024-04-24T23:04:46.831 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52523.log 2024-04-24T23:04:46.832 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52523.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52523.log.gz 2024-04-24T23:04:46.832 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83441.log 2024-04-24T23:04:46.833 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.83441.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.83441.log.gz 2024-04-24T23:04:46.833 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81880.log 2024-04-24T23:04:46.834 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.81880.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.81880.log.gz 2024-04-24T23:04:46.834 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56712.log 2024-04-24T23:04:46.835 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56712.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56712.log.gz 2024-04-24T23:04:46.835 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52503.log 2024-04-24T23:04:46.838 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.52503.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.52503.log.gz 2024-04-24T23:04:46.838 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72535.log 2024-04-24T23:04:46.838 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.72535.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.72535.log.gz 2024-04-24T23:04:46.838 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48598.log 2024-04-24T23:04:46.839 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.48598.log: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.48598.log.gz 2024-04-24T23:04:46.839 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74465.log 2024-04-24T23:04:46.840 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.74465.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74465.log.gz 2024-04-24T23:04:46.840 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50338.log 2024-04-24T23:04:46.841 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.50338.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.50338.log.gz 2024-04-24T23:04:46.841 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65400.log 2024-04-24T23:04:46.842 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65400.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65400.log.gz 2024-04-24T23:04:46.842 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63498.log 2024-04-24T23:04:46.843 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.63498.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63498.log.gz 2024-04-24T23:04:46.843 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66500.log 2024-04-24T23:04:46.844 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.66500.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66500.log.gz 2024-04-24T23:04:46.845 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60215.log 2024-04-24T23:04:46.845 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60215.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.60215.log.gz 2024-04-24T23:04:46.845 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75359.log 2024-04-24T23:04:46.848 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75359.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75359.log.gz 2024-04-24T23:04:46.848 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64594.log 2024-04-24T23:04:46.849 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.64594.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64594.log.gz 2024-04-24T23:04:46.849 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71550.log 2024-04-24T23:04:46.850 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.71550.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71550.log.gz 2024-04-24T23:04:46.850 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59617.log 2024-04-24T23:04:46.851 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.59617.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59617.log.gz 2024-04-24T23:04:46.851 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61608.log 2024-04-24T23:04:46.852 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61608.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61608.log.gz 2024-04-24T23:04:46.852 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76843.log 2024-04-24T23:04:46.855 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76843.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76843.log.gz 2024-04-24T23:04:46.855 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61826.log 2024-04-24T23:04:46.857 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.61826.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61826.log.gz 2024-04-24T23:04:46.857 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65537.log 2024-04-24T23:04:46.859 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.65537.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69067.log 2024-04-24T23:04:46.859 INFO:teuthology.orchestra.run.smithi134.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65537.log.gz 2024-04-24T23:04:46.861 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.69067.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69067.log.gz 2024-04-24T23:04:46.861 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50504.log 2024-04-24T23:04:46.862 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.50504.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50504.log.gz 2024-04-24T23:04:46.863 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75669.log 2024-04-24T23:04:46.865 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75669.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75669.log.gz 2024-04-24T23:04:46.866 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70124.log 2024-04-24T23:04:46.866 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56463.log 2024-04-24T23:04:46.867 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.56463.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56463.log.gz 2024-04-24T23:04:46.867 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60779.log 2024-04-24T23:04:46.868 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60779.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60779.log.gz 2024-04-24T23:04:46.868 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78891.log 2024-04-24T23:04:46.869 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.78891.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.78891.log.gz 2024-04-24T23:04:46.869 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71200.log 2024-04-24T23:04:46.870 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.71200.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71200.log.gz 2024-04-24T23:04:46.870 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63840.log 2024-04-24T23:04:46.871 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.63840.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63840.log.gz 2024-04-24T23:04:46.871 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78553.log 2024-04-24T23:04:46.872 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.78553.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.78553.log.gz 2024-04-24T23:04:46.872 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64642.log 2024-04-24T23:04:46.873 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.64642.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64642.log.gz 2024-04-24T23:04:46.873 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46681.log 2024-04-24T23:04:46.874 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.46681.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.46681.log.gz 2024-04-24T23:04:46.874 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67828.log 2024-04-24T23:04:46.875 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.67828.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.67828.log.gz 2024-04-24T23:04:46.875 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74631.log 2024-04-24T23:04:46.876 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.74631.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74631.log.gz 2024-04-24T23:04:46.876 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82493.log 2024-04-24T23:04:46.877 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.82493.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53170.log.gz 2024-04-24T23:04:46.877 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70527.log 2024-04-24T23:04:46.877 INFO:teuthology.orchestra.run.smithi134.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.82493.log.gz 2024-04-24T23:04:46.878 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70124.log: /var/log/ceph/cluster1-client.mirror.70527.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70124.log.gz 2024-04-24T23:04:46.878 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78254.log 2024-04-24T23:04:46.879 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.78254.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70527.log.gz 2024-04-24T23:04:46.879 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69648.log 2024-04-24T23:04:46.880 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69648.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69648.log.gz 2024-04-24T23:04:46.880 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83882.log 2024-04-24T23:04:46.881 INFO:teuthology.orchestra.run.smithi134.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.78254.log.gz 2024-04-24T23:04:46.881 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.74465.log 2024-04-24T23:04:46.881 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.83882.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.83882.log.gz 2024-04-24T23:04:46.882 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.2.74465.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65468.log 2024-04-24T23:04:46.882 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.65468.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65468.log.gz 2024-04-24T23:04:46.882 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74607.log 2024-04-24T23:04:46.884 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.74607.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74607.log.gz 2024-04-24T23:04:46.884 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64493.log 2024-04-24T23:04:46.886 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.64493.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64493.log.gz 2024-04-24T23:04:46.886 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51610.log 2024-04-24T23:04:46.887 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.51610.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51610.log.gz 2024-04-24T23:04:46.887 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73535.log 2024-04-24T23:04:46.888 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.73535.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.2.74465.log.gz 2024-04-24T23:04:46.888 INFO:teuthology.orchestra.run.smithi134.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73535.log.gz 2024-04-24T23:04:46.888 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55056.log 2024-04-24T23:04:46.889 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.55056.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55056.log.gz 2024-04-24T23:04:46.889 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55675.log 2024-04-24T23:04:46.902 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55675.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55675.log.gz 2024-04-24T23:04:46.902 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67406.log 2024-04-24T23:04:46.903 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.67406.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.67406.log.gz 2024-04-24T23:04:46.903 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75525.log 2024-04-24T23:04:46.904 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75525.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75525.log.gz 2024-04-24T23:04:46.904 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75070.log 2024-04-24T23:04:46.906 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75070.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75070.log.gz 2024-04-24T23:04:46.906 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68526.log 2024-04-24T23:04:46.907 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.68526.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68526.log.gz 2024-04-24T23:04:46.907 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65206.log 2024-04-24T23:04:46.908 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.65206.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65206.log.gz 2024-04-24T23:04:46.908 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64618.log 2024-04-24T23:04:46.909 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.64618.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64618.log.gz 2024-04-24T23:04:46.910 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54542.log 2024-04-24T23:04:46.911 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.54542.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54542.log.gz 2024-04-24T23:04:46.911 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80601.log 2024-04-24T23:04:46.912 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80601.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80601.log.gz 2024-04-24T23:04:46.912 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62683.log 2024-04-24T23:04:46.914 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.62683.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62683.log.gz 2024-04-24T23:04:46.915 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73273.log 2024-04-24T23:04:46.916 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.73273.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73273.log.gz 2024-04-24T23:04:46.916 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75628.log 2024-04-24T23:04:46.917 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.75628.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75628.log.gz 2024-04-24T23:04:46.917 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-04-24T23:04:46.921 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2.audit.log: 92.1% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-04-24T23:04:46.921 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83732.log 2024-04-24T23:04:46.923 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83732.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.83732.log.gz 2024-04-24T23:04:46.923 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68288.log 2024-04-24T23:04:46.924 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.68288.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68288.log.gz 2024-04-24T23:04:46.924 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62023.log 2024-04-24T23:04:46.925 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.62023.log: 2.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62023.log.gz 2024-04-24T23:04:46.925 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70248.log 2024-04-24T23:04:46.926 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.70248.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70248.log.gz 2024-04-24T23:04:46.926 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.83977.log 2024-04-24T23:04:46.930 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.2.83977.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.2.83977.log.gz 2024-04-24T23:04:46.930 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48615.log 2024-04-24T23:04:46.931 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.48615.log: 80.0% -- replaced with /var/log/ceph/cluster1-client.admin.48615.log.gz 2024-04-24T23:04:46.931 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55804.log 2024-04-24T23:04:46.937 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55804.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55804.log.gz 2024-04-24T23:04:46.937 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67780.log 2024-04-24T23:04:46.938 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.67780.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.67780.log.gz 2024-04-24T23:04:46.938 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55910.log 2024-04-24T23:04:46.944 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.55910.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55910.log.gz 2024-04-24T23:04:46.945 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51973.log 2024-04-24T23:04:46.946 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.51973.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51973.log.gz 2024-04-24T23:04:46.946 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80578.log 2024-04-24T23:04:46.947 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80578.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.80578.log.gz 2024-04-24T23:04:46.947 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62356.log 2024-04-24T23:04:46.948 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.62356.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62356.log.gz 2024-04-24T23:04:46.948 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-04-24T23:04:46.959 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62880.log 2024-04-24T23:04:46.971 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.62880.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62880.log.gz 2024-04-24T23:04:46.971 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55366.log 2024-04-24T23:04:46.983 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55366.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55452.log 2024-04-24T23:04:46.988 INFO:teuthology.orchestra.run.smithi134.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55366.log.gz 2024-04-24T23:04:46.999 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.55452.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53455.log 2024-04-24T23:04:46.999 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.83853.log.gz 2024-04-24T23:04:46.999 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55452.log.gz 2024-04-24T23:04:47.004 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.53455.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53455.log.gz 2024-04-24T23:04:47.004 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66428.log 2024-04-24T23:04:47.005 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.66428.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66428.log.gz 2024-04-24T23:04:47.005 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60663.log 2024-04-24T23:04:47.006 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60663.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60663.log.gz 2024-04-24T23:04:47.006 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80173.log 2024-04-24T23:04:47.007 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80173.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80173.log.gz 2024-04-24T23:04:47.007 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76492.log 2024-04-24T23:04:47.009 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76492.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76492.log.gz 2024-04-24T23:04:47.009 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.59491.log 2024-04-24T23:04:47.011 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.0.59491.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.59491.log.gz 2024-04-24T23:04:47.012 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51285.log 2024-04-24T23:04:47.013 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.51285.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.51285.log.gz 2024-04-24T23:04:47.013 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48717.log 2024-04-24T23:04:47.014 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.48717.log: 58.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48717.log.gz 2024-04-24T23:04:47.014 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.53287.log 2024-04-24T23:04:47.020 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.2.53287.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57928.log.gz 2024-04-24T23:04:47.020 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.2.53287.log.gz 2024-04-24T23:04:47.020 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54161.log 2024-04-24T23:04:47.021 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.54161.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54161.log.gz 2024-04-24T23:04:47.021 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85203.log 2024-04-24T23:04:47.022 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.admin.85203.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85203.log.gz 2024-04-24T23:04:47.022 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53126.log 2024-04-24T23:04:47.027 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.53126.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60012.log.gz 2024-04-24T23:04:47.027 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53126.log.gz 2024-04-24T23:04:47.028 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67517.log 2024-04-24T23:04:47.028 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76719.log 2024-04-24T23:04:47.029 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67517.log: /var/log/ceph/cluster2-client.mirror.76719.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67517.log.gz 2024-04-24T23:04:47.031 INFO:teuthology.orchestra.run.smithi134.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76719.log.gz 2024-04-24T23:04:47.031 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.81667.log 2024-04-24T23:04:47.031 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56383.log 2024-04-24T23:04:47.033 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.1.81667.log: /var/log/ceph/cluster2-client.mirror.56383.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56383.log.gz 2024-04-24T23:04:47.034 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77965.log 2024-04-24T23:04:47.034 INFO:teuthology.orchestra.run.smithi134.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.81667.log.gz 2024-04-24T23:04:47.035 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.77965.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77965.log.gz 2024-04-24T23:04:47.035 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73104.log 2024-04-24T23:04:47.035 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84932.log 2024-04-24T23:04:47.037 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73104.log: /var/log/ceph/cluster1-client.mirror.84932.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84932.log.gz 2024-04-24T23:04:47.037 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71152.log 2024-04-24T23:04:47.037 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.71152.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71152.log.gz 2024-04-24T23:04:47.037 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76513.log 2024-04-24T23:04:47.037 INFO:teuthology.orchestra.run.smithi134.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73104.log.gz 2024-04-24T23:04:47.038 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.76513.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.76513.log.gz 2024-04-24T23:04:47.038 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79161.log 2024-04-24T23:04:47.038 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83773.log 2024-04-24T23:04:47.039 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.79161.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79161.log.gz 2024-04-24T23:04:47.041 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83773.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.83773.log.gz 2024-04-24T23:04:47.041 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67237.log 2024-04-24T23:04:47.041 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76864.log 2024-04-24T23:04:47.043 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67237.log: /var/log/ceph/cluster2-client.mirror.76864.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.76864.log.gz 2024-04-24T23:04:47.043 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64078.log 2024-04-24T23:04:47.043 INFO:teuthology.orchestra.run.smithi134.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67237.log.gz 2024-04-24T23:04:47.044 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.64078.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64078.log.gz 2024-04-24T23:04:47.045 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58908.log 2024-04-24T23:04:47.045 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63903.log 2024-04-24T23:04:47.045 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.58908.log: /var/log/ceph/cluster2-client.mirror.63903.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58908.log.gz 2024-04-24T23:04:47.046 INFO:teuthology.orchestra.run.smithi134.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63903.log.gz 2024-04-24T23:04:47.046 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69906.log 2024-04-24T23:04:47.047 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79386.log 2024-04-24T23:04:47.047 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.69906.log: /var/log/ceph/cluster1-client.mirror.79386.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69906.log.gz 2024-04-24T23:04:47.048 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.79386.log.gz 2024-04-24T23:04:47.048 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70890.log 2024-04-24T23:04:47.048 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83208.log 2024-04-24T23:04:47.048 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.70890.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70890.log.gz 2024-04-24T23:04:47.051 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.83208.log: 82.0% -- replaced with /var/log/ceph/cluster2-client.mirror.83208.log.gz 2024-04-24T23:04:47.051 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57376.log 2024-04-24T23:04:47.052 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84553.log 2024-04-24T23:04:47.052 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.57376.log: /var/log/ceph/cluster1-client.mirror.84553.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57376.log.gz 2024-04-24T23:04:47.053 INFO:teuthology.orchestra.run.smithi134.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.84553.log.gz 2024-04-24T23:04:47.053 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67358.log 2024-04-24T23:04:47.053 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64054.log 2024-04-24T23:04:47.054 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.67358.log: /var/log/ceph/cluster1-client.mirror.64054.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67358.log.gz 2024-04-24T23:04:47.054 INFO:teuthology.orchestra.run.smithi134.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64054.log.gz 2024-04-24T23:04:47.054 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70846.log 2024-04-24T23:04:47.054 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52922.log 2024-04-24T23:04:47.055 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.70846.log: /var/log/ceph/cluster2-client.mirror.52922.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52922.log.gz 2024-04-24T23:04:47.055 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73681.log 2024-04-24T23:04:47.056 INFO:teuthology.orchestra.run.smithi134.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70846.log.gz 2024-04-24T23:04:47.057 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.73681.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73681.log.gz 2024-04-24T23:04:47.057 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.57520.log 2024-04-24T23:04:47.057 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56951.log 2024-04-24T23:04:47.060 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.0.57520.log: /var/log/ceph/cluster2-client.mirror.56951.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56951.log.gz 2024-04-24T23:04:47.060 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60130.log 2024-04-24T23:04:47.060 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.60130.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.0.57520.log.gz 2024-04-24T23:04:47.060 INFO:teuthology.orchestra.run.smithi134.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60130.log.gz 2024-04-24T23:04:47.060 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80854.log 2024-04-24T23:04:47.060 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.60573.log 2024-04-24T23:04:47.073 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.80854.log: /var/log/ceph/cluster2-client.mirror.3.60573.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68910.log 2024-04-24T23:04:47.074 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.3.60573.log.gz 2024-04-24T23:04:47.075 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-client.mirror.68910.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68910.log.gz 2024-04-24T23:04:47.075 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54566.log 2024-04-24T23:04:47.076 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.54566.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54566.log.gz 2024-04-24T23:04:47.076 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72487.log 2024-04-24T23:04:47.077 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80736.log 2024-04-24T23:04:47.078 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.72487.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.72487.log.gz 2024-04-24T23:04:47.078 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.mirror.80736.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80736.log.gz 2024-04-24T23:04:47.229 INFO:teuthology.orchestra.run.smithi134.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61284.log.gz 2024-04-24T23:04:47.449 INFO:teuthology.orchestra.run.smithi134.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-04-24T23:04:47.583 INFO:teuthology.orchestra.run.smithi134.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.80854.log.gz 2024-04-24T23:04:49.515 INFO:teuthology.orchestra.run.smithi080.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-04-24T23:04:50.402 INFO:teuthology.orchestra.run.smithi080.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-04-24T23:04:50.827 INFO:teuthology.orchestra.run.smithi080.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-04-24T23:04:50.830 INFO:teuthology.orchestra.run.smithi080.stderr: 2024-04-24T23:04:50.830 INFO:teuthology.orchestra.run.smithi080.stderr:real 0m8.341s 2024-04-24T23:04:50.830 INFO:teuthology.orchestra.run.smithi080.stderr:user 0m23.739s 2024-04-24T23:04:50.830 INFO:teuthology.orchestra.run.smithi080.stderr:sys 0m0.928s 2024-04-24T23:05:36.553 INFO:teuthology.orchestra.run.smithi134.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-04-24T23:05:36.935 INFO:teuthology.orchestra.run.smithi134.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-04-24T23:05:50.682 INFO:teuthology.orchestra.run.smithi134.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-04-24T23:05:50.683 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-04-24T23:05:50.683 INFO:teuthology.orchestra.run.smithi134.stderr:real 1m8.151s 2024-04-24T23:05:50.683 INFO:teuthology.orchestra.run.smithi134.stderr:user 3m9.925s 2024-04-24T23:05:50.684 INFO:teuthology.orchestra.run.smithi134.stderr:sys 0m8.131s 2024-04-24T23:05:50.685 INFO:tasks.ceph:Archiving logs... 2024-04-24T23:05:50.685 DEBUG:teuthology.misc:Transferring archived files from smithi080:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-24_21:25:53-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7672255/remote/smithi080/log 2024-04-24T23:05:50.907 DEBUG:teuthology.orchestra.run.smithi080:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T23:05:54.938 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-24_21:25:53-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7672255/remote/smithi134/log 2024-04-24T23:05:54.939 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T23:06:26.794 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-24T23:06:26.808 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-24T23:06:26.808 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T23:06:26.811 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T23:06:26.901 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-24T23:06:26.914 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-24T23:06:26.914 DEBUG:teuthology.orchestra.run.smithi080:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T23:06:26.917 DEBUG:teuthology.orchestra.run.smithi134:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T23:06:26.933 INFO:teuthology.orchestra.run.smithi080.stderr:bash: ntpq: command not found 2024-04-24T23:06:26.937 INFO:teuthology.orchestra.run.smithi134.stderr:bash: ntpq: command not found 2024-04-24T23:06:26.938 INFO:teuthology.orchestra.run.smithi080.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T23:06:26.938 INFO:teuthology.orchestra.run.smithi080.stdout:=============================================================================== 2024-04-24T23:06:26.938 INFO:teuthology.orchestra.run.smithi080.stdout:^- hv01.front.sepia.ceph.com 2 6 377 29 +1270us[+1270us] +/- 582ms 2024-04-24T23:06:26.938 INFO:teuthology.orchestra.run.smithi080.stdout:^* hv02.front.sepia.ceph.com 3 8 377 95 +39us[ +17us] +/- 79ms 2024-04-24T23:06:26.939 INFO:teuthology.orchestra.run.smithi080.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 -48us[ -48us] +/- 72ms 2024-04-24T23:06:26.939 INFO:teuthology.orchestra.run.smithi080.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:06:26.941 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T23:06:26.942 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-04-24T23:06:26.942 INFO:teuthology.orchestra.run.smithi134.stdout:^- hv01.front.sepia.ceph.com 2 6 377 31 +1278us[+1278us] +/- 582ms 2024-04-24T23:06:26.942 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 33 +43us[ +35us] +/- 80ms 2024-04-24T23:06:26.942 INFO:teuthology.orchestra.run.smithi134.stdout:^* hv03.front.sepia.ceph.com 3 6 377 33 -42us[ -50us] +/- 72ms 2024-04-24T23:06:26.942 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:06:26.943 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-24T23:06:26.955 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-24T23:06:26.956 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-24T23:06:26.972 DEBUG:teuthology.orchestra.run.smithi080:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T23:06:27.047 DEBUG:teuthology.orchestra.run.smithi134:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T23:06:27.116 DEBUG:teuthology.orchestra.run.smithi080:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-24T23:06:27.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:06:27.147 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-24T23:06:27.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:06:27.178 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-24T23:06:27.191 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-24T23:06:27.203 INFO:teuthology.task.internal:Duration was 2915.615359 seconds 2024-04-24T23:06:27.203 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-24T23:06:27.215 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-24T23:06:27.216 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T23:06:27.218 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T23:06:27.264 INFO:teuthology.orchestra.run.smithi080.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T23:06:27.266 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T23:06:27.572 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-24T23:06:27.572 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi080.front.sepia.ceph.com 2024-04-24T23:06:27.572 DEBUG:teuthology.orchestra.run.smithi080:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T23:06:27.595 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi134.front.sepia.ceph.com 2024-04-24T23:06:27.595 DEBUG:teuthology.orchestra.run.smithi134:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T23:06:27.617 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-24T23:06:27.618 DEBUG:teuthology.orchestra.run.smithi080:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T23:06:27.638 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T23:06:27.700 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-24T23:06:27.700 DEBUG:teuthology.orchestra.run.smithi080:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T23:06:27.722 DEBUG:teuthology.orchestra.run.smithi134:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T23:06:27.813 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-24T23:06:27.828 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-24T23:06:27.828 DEBUG:teuthology.orchestra.run.smithi080:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T23:06:27.856 DEBUG:teuthology.orchestra.run.smithi134:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T23:06:27.883 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-24T23:06:27.897 DEBUG:teuthology.orchestra.run.smithi080:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T23:06:27.899 DEBUG:teuthology.orchestra.run.smithi134:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T23:06:27.923 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern = core 2024-04-24T23:06:27.949 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = core 2024-04-24T23:06:27.964 DEBUG:teuthology.orchestra.run.smithi080:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T23:06:28.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:06:28.021 DEBUG:teuthology.orchestra.run.smithi134:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T23:06:28.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:06:28.038 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-24T23:06:28.052 INFO:teuthology.task.internal:Transferring archived files... 2024-04-24T23:06:28.053 DEBUG:teuthology.misc:Transferring archived files from smithi080:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-24_21:25:53-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7672255/remote/smithi080 2024-04-24T23:06:28.053 DEBUG:teuthology.orchestra.run.smithi080:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T23:06:28.102 DEBUG:teuthology.misc:Transferring archived files from smithi134:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-24_21:25:53-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7672255/remote/smithi134 2024-04-24T23:06:28.102 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T23:06:28.142 INFO:teuthology.task.internal:Removing archive directory... 2024-04-24T23:06:28.142 DEBUG:teuthology.orchestra.run.smithi080:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T23:06:28.144 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T23:06:28.193 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-24T23:06:28.208 INFO:teuthology.task.internal:Not uploading archives. 2024-04-24T23:06:28.209 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-24T23:06:28.222 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-24T23:06:28.223 DEBUG:teuthology.orchestra.run.smithi080:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T23:06:28.225 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T23:06:28.240 INFO:teuthology.orchestra.run.smithi080.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 23:06 /home/ubuntu/cephtest 2024-04-24T23:06:28.250 INFO:teuthology.orchestra.run.smithi134.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 23:06 /home/ubuntu/cephtest 2024-04-24T23:06:28.252 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-24T23:06:28.265 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-24T23:06:28.297 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} duration: 2915.6153588294983 flavor: default owner: scheduled_yuriw@teuthology success: true 2024-04-24T23:06:28.298 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T23:06:28.380 INFO:teuthology.run:pass